DEV Community: A. Sharif

Notes on PostgreSQL: Common Table Expressions

A. Sharif — Mon, 20 Jun 2022 19:30:11 +0000

Introduction

You might already have heard about common table expressions, in short CTEs - if not, these notes might also be helpful for furthering your understanding on the topic. It's one of these very nice features that come with working with PostgreSQL, especially if you were switching over from MySQL prior to version 8.0. These notes will help in understanding why CTEs are sometimes referred to as "WITH queries", how to leverage multiple CTEs and even a little bit of recursion.

Basics

To get a good understanding of the very basics of CTEs, let's begin with the syntax. To construct a common table expression we leverage the WITH keyword.
NOTE: CTEs are also referred to as WITH queries due to the WITH keyword they are based on.
Consulting the PostgreSQL documentation we can see what a basic structure of a CTE looks like:

WITH CTE_NAME (OPTIONAL_COLUMN_NAMES) AS (CTE_SELECT_QUERY)
SELECT * FROM CTE_NAME;

The WITH clause is followed by the name of the CTE and an optional list of column names. Finally the selection query is defined after the AS keyword.
Once the CTE is defined, it can be accessed in any SELECT, INSERT, UPDATE and even DELETE statement.

Let's take a look at an example to get a better idea. The following examples assume we have a list of vinyl records, including the artist name, year of release, label and format. Furthermore we have a list of all records sold, including date of sale and the corresponding price. Check the schema here.
In the first example we would like to find the most sold records via querying the sales table:

WITH most_sold_records (record_id, sold_items) AS(
  SELECT   record,
           count(id)
  FROM     sales
  GROUP BY record
)

SELECT   *
FROM     most_sold_records
ORDER BY sold_items DESC, record_id ASC;

The above example demonstrates how to construct a CTE and then accessing it in the follow-up statement. The result of the query is a list containing any items sold, ordered by the number of items sold. From the query we can see that we have information about the record id, so our next step would be to join the newly created CTE with the records table, to display information like title, label and format in addition to the number of sold items.

WITH most_sold_records (record_id, sold_items) AS(
  SELECT   record,
           count(id)
  FROM     sales
  GROUP BY record
)

SELECT     record_id,
           name,
           artist,
           label,
           format,
           sold_items
FROM       most_sold_records
INNER JOIN records ON records.id = record_id
ORDER BY   record_id DESC, sold_items ASC;

Let's take a look at another basic example before getting into more advanced territory. Our vinyl catalog is comprised of different formats like 12", 10" or 7" single vinyl or single or double LPs (albums). We would like to retrieve a list where a vinyl record should either be defined as a single or album.

WITH record_catalog AS(
  SELECT id,
         name,
         label,
         CASE
           WHEN format IN ('12', '10', '7')
            THEN 'Single'
           ELSE 'Album'
         END AS type
  FROM   records
)

SELECT * from record_catalog;

This would return a list of vinyl records where type is either a single record or an album. Now we can use the record_catalog CTE to filter out all singles for example.

WITH record_catalog AS(
  SELECT id,
         name,
         label,
         CASE
           WHEN format IN ('12', '10', '7')
             THEN 'Single'
           ELSE 'Album'
         END AS type
  FROM   records
)

SELECT *
FROM   record_catalog
WHERE  type = 'Album';

Up until this point, we should now have a very good understanding of the basic structure of common table expressions in PostgreSQL. In the following section we will try to see what further possible combinations are possible.

Advanced

The WITH clause accepts multiple CTEs:

WITH 
    CTE_NAME_1 (OPTIONAL_COLUMN_NAMES) AS (CTE_SELECT_QUERY_1),
    CTE_NAME_2 (OPTIONAL_COLUMN_NAMES) AS (CTE_SELECT_QUERY_2)

SELECT * FROM CTE_NAME_1 
UNION 
SELECT * FROM CTE_NAME_2

Being able to define multiple CTEs opens up more possibilities to leverage. For example we might want to display a list showing the number of units and the highest and lowest price sold per record as well as the highest price a record was sold by that artist.
The first step is to break it up into two queries. A regular query for finding the max, min and count for every record inside the sales table:

SELECT   count(*) AS total_units_sold, 
         MAX(price) AS highest_price, 
         MIN(price) AS lowest_price,
         a.id As artist_id,
         a.name As artist_name,
         r.name AS record_title
FROM     sales 
JOIN     records r ON (record = r.id)
JOIN     artists a ON (r.artist = a.id)
GROUP BY r.name, a.id;

This would return us a list with all the needed record sales information. For the second part of the query we can change the GROUP BY column to retrieve the highest price on an artist basis.

SELECT     MAX(price) AS highest_overall_price,
           a.id AS artist_id
FROM       sales
INNER JOIN records r ON (record = r.id)
INNER JOIN artists a ON (r.artist = a.id)
GROUP BY   a.id
ORDER BY   a.id;

What is left is now to take these two queries, convert them to CTEs and then join them together.

WITH record_sales AS (
  -- Find the amount sold, highest and lowest price per record
  SELECT     COUNT(*) AS total_units_sold, 
             MAX(price) AS highest_price, 
             MIN(price) AS lowest_price,
             a.id As artist_id,
             a.name As artist_name,
             r.name AS record_title
  FROM       sales 
  INNER JOIN records r ON (record = r.id)
  INNER JOIN artists a ON (r.artist = a.id)
  GROUP BY   r.name, a.id
  ORDER BY   a.id 
),
artist_sales AS (
  -- Find the highest price per artist
  SELECT     MAX(price) AS highest_overall_price,
             a.id AS artist_id
  FROM       sales
  INNER JOIN records r ON (record = r.id)
  INNER JOIN artists a ON (r.artist = a.id)
  GROUP BY   a.id
  ORDER BY   a.id
)

SELECT       * 
FROM         record_sales
INNER JOIN   artist_sales USING (artist_id);

One more interesting aspect, if we recall the format example we previously created, is that by defining a record to either be of type Single or Album, we could extend it to filter out based on said type in a second CTE. The means that we can access CTEs within other CTEs, check the following adaption:

WITH record_catalog AS(
  SELECT id,
         name,
         label,
         CASE
            WHEN format IN ('12', '10', '7')
             THEN 'Single'
            ELSE 'Album'
          END
         AS type
  FROM records
),
album AS (SELECT *
          FROM record_catalog
          WHERE type = 'Album')

SELECT *
FROM   album;

Recursive Common Table Expressions

Sometimes we need to work through some data that has some sort of hierarchy and we would like to go through the data starting from the root and then traversing level by level, finding references to the previous level. For example in our online vinyl record store, we have an artists table that contains a field alias. The alias is used when an artist also uses a different name and sometimes even this alias might have another alias associated with it.
Normally, to find the root artist, we would check to see which artist entries have no alias references.

SELECT *
FROM   artists
WHERE  alias IS NULL;

The standard definition of a WITH RECURSIVE in PostgreSQL has the following structure:

WITH RECURSIVE CTE_NAME (OPTIONAL_COLUMN_NAMES) AS(
    CTE_NON_RECURSIVE_SELECT_QUERY
    UNION ALL
    CTE_RECURSIVE_SELECT_QUERY
)
SELECT * FROM CTE_NAME;

The documentation summarises this nicely: "The general form of a recursive WITH query is always a non-recursive term, then UNION (or UNION ALL), then a recursive term, where only the recursive term can contain a reference to the query's own output."
So we can derive that the non-recursive term is the base result set and the the recursive term, which can consist of one to n CTEs are joined by an UNION [ALL]. The recursion halts if the previous iteration doesn't return any new rows.

Now that we have a basic idea of the structure, the next step is to convert this top level artist lookup into a CTE:

WITH RECURSIVE root_aritst AS (
  SELECT *
  FROM   artists
  WHERE  alias IS NULL
)

SELECT * FROM root_aritst;

By using the WITH RECURSIVE clause we can rebuild our query to go through level by level and return a hierarchical structure representing an artist's aliases and the aliases for these aliases etc.

WITH RECURSIVE leveln AS (
  SELECT *, '' AS alias
  FROM   artists
  WHERE  alias IS NULL
  UNION ALL
  SELECT artists.*, leveln.name alias FROM artists
  JOIN leveln on leveln.id = artists.alias
)

SELECT * FROM leveln;

We start with the root level, the base artist representations, and then start looking for artist entries that reference these representations. Once we have the 1st level aliases, our query will try to find all aliases referencing these 1st level aliases and so forth. By designing our query this way, we can go n levels deep.

NOTE: We need to ensure that we don't run into some endless recursion.
A follow-up post will focus on optimisations and how to avoid pitfalls.

Summary

These notes on CTEs should have helped with better understanding common table expressions. Especially how a CTE is constructed and how to combine multiple CTEs for more complex operations as well as how to use recursion for specific cases.
As a round up, we can see that CTEs come with some nice benefits:

They can be more readable than regular subqueries and very helpful when it comes to reuse.
Multiple CTEs can be constructed and reused and combined in a later statement, enabling us to use the expressions in a more flexible manner.
Leveraging recursive CTEs, we can do more complicated calculations while keeping these complex operations readable.
CTEs can be joined with other tables and window functions. This can help us to filter out specific data step by step. Refining our result set along the way.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Notes on Advanced TypeScript: Runtime Validations

A. Sharif — Wed, 04 May 2022 14:24:31 +0000

Introduction

These notes should help in better understanding advanced TypeScript topics and might be helpful when needing to lookup up how to leverage TypeScript in a specific situation. All examples are based on TypeScript 4.6.

Note: This post is an update version of the original Notes on TypeScript: Handling Side-Effects

Basic

There are situations when working with TypeScript, where we can't guarantee that the types reflect the actual data we are working with. Examples for these types of situations include reading from a file, fetching data from an external endpoint or loading information saved in local storage. In all of the above scenarios we can't guarantee that the data entering our application actually reflects the types we defined. Further more, in any of these scenarios we can be running into runtime errors, no matter what the type actually claims.
This means once we're dealing with external data, that is not defined at compile time, we need some mechanism to safely handle this type of data.

To make it more practical, let's assume the following situation: we want to load a user from a pre-defined endpoint.

const loadUser = (id: number) => {
  fetch(`http://www.your-defined-endpoint.com/users/${id}`)
    .then((response) => response.json())
    .then((user: User) => saveUser(user))
    .catch((error) => {
      console.log({ error });
    });
};

At first look this all sounds reasonable, we fetch a user by id, and then save the data for further processing. If you take a closer look at the code, you will notice that we defined the data to be of type User after decoding the json data. The User type in this example is defined as follows:

type User = {
  id: number;
  name: string;
  active: boolean;
  profile: {
    activatedAt: number;
  };
};

Interestingly the code will compile and TypeScript will show no errors as we defined a User and claimed that the response, once decoded, will always be of aforementioned type. Even more interesting is the fact that calling the json function on the response object returns an Promise<any>, so there is no actual guarantee that we are dealing with a User type at runtime.

Let's see a scenario where our assumptions might fail, so let's add a saveUser function, that expects a user with some profile information:

const saveUser = (user: User) => {
  const activationDate = user.profile.activatedAt;
  // do something with the information...
};

Now how can our application break? The code above will compile, but what happens when the returned user object doesn't have any profile information? Let's assume that at runtime, we suddenly receive the following object:

{
  id: 1,
  name: "Some User Name",
  active: true,
  extended: {
      activatedAt: 1640995200000
  }
};

The result will still be a User inside our application, but we will run into an error at runtime, as soon as we call the saveUser function. One way to deal with this, is to get more defensive, by exteding our function to check if the property profile even exists:

const saveUser = (user: User) => {
  if (user && user.profile && user.profile.activatedAt) {
    const activationDate = user.profile.activatedAt;
    // do something with the information...
  } else {
    // do something else
  }
};

But this will quickly become complicated when we have to do these checks all over our application when working with external data. Rather, we want to do this check as early as possible, in fact at the moment we have access to said data.

Advanced

TypeScript doesn't offer any runtime JSON validation capabilities, but there are libraries in the TypeScript eco-system that we can leverage for that specific case.
We will use the popular io-ts library to ensure the data we are working on is reliable throught the application. Our approach will be to decode any external data entering our application.

io-ts is written by Giulio Canti and offers runtime type validations. For more information on io-ts consult the README. So called codecs are used to encode/decode data.These codecs are runtime representations of specific static types and can be composed to build even larger type validations.

Codecs enable us to encode and decode any in/out data and the built-in decode method returns an Either type, which represents success (Right) and failure (Left). Via leveraging this functionality we can decode external data and handle the success/failure case specifically. To get a better understanding let's rebuild our previous example using the io-ts library.

import * as t from "io-ts";

const User = t.type({
  id: t.number,
  name: t.string,
  active: t.boolean,
  profile: t.type({
    activatedAt: t.number,
  }),
});

By combing different codecs like string or number we can construct a User runtime type, that we can use for validating any incoming user data.

The previous basic construct has the same shape as the User type we defined previously. What we don't want though, is to redefine the User as a static type as well. io-ts can help us here, by offering TypeOf which enables user land to generate a static representation of the constructed User.

type UserType = t.TypeOf<typeof User>;

Interestingly this will give us the same representation we defined in the beginning:

type UserType = {
  id: number,
  name: string,
  active: boolean,
  profile: {
    activatedAt: number,
  },
};

Once we have a defined shape, we can verify if the data is of that expected shape and either handle the success or failure case:

const userA = {
  id: 1,
  name: "Test User A",
  active: true,
  profile: {
    activatedAt: t.number,
  },
};

const result = User.decode(userA);

if (result._tag === "Right") {
  // handle the success case
  // access the data
  result.right;
} else {
  // handle the failure
}

The result of the decode function contains a _tag property that can either be a Right or Left string, which represent success or failure. Furthermore we have access to a right and left property, containing the decoded data in the success case (right) or an error message in the failure case (right).
The above example can be extended to use a so called PathReporter for error message handling:

import { PathReporter } from "io-ts/lib/PathReporter";

if (result._tag === "Right") {
  // handle the success case
  // access the data
  result.right;
} else {
  // handle the failure
  console.warn(PathReporter.report(result).join("\n"));
}

io-ts also comes with fp-ts as a peer dependency, which offers useful utility functions like isRight or fold. We can use the the isRight function to check if the decoded result is valid, instead of having to manually handle this via the _tag property.

import * as t from "io-ts";
import { isRight } from "fp-ts/lib/Either";

const userA = {
  id: 1,
  name: "Test User A",
  active: true,
  profile: {
    activatedAt: t.number,
  },
};

isRight(User.decode(userA)); // true

const userB = {
  id: 1,
  name: "Test User",
  active: true,
  extended: {
    activatedAt: t.number,
  },
};

isRight(User.decode(userB)); // false

One more useful functionality that will help us when working with the Either type, that the decode returns is fold, which enables us to define a success and failure path, check the following example for more clarification:

const validate = fold(
  (error) => console.log({ error }),
  (result) => console.log({ result })
);

// success case
validate(User.decode(userA));

// failure case
validate(User.decode(userB));

Using fold enables us to handle valid or invalid data when calling our fetch functionality. The loadUser function could now be refactored to handle these cases.

const resolveUser = fold(
  (errors: t.Errors) => {
    throw new Error(`${errors.length} errors found!`);
  },
  (user: User) => saveUser(user)
);

const loadUser = (id: number) => {
  fetch(`http://www.your-defined-endpoint.com/users/${id}`)
    .then((response) => response.json())
    .then((user) => resolveUser(User.decode(user)))
    .catch((error) => {
      console.log({ error });
    });
};

We might handle any incorrect representation by throwing another error. This prevents the data from being passed around in our application. There are more improvement we can make here. Right now, we're being very specific in how we're handling the User decoding. There might be an opportunity to write a general function that handles any promise based data.

const decodePromise = <I, O>(type: t.Decoder<I, O>, value: I): Promise<O> => {
  return (
    fold < t.Errors,
    O,
    Promise <
      O >>
        ((errors) => Promise.reject(errors),
        (result) => Promise.resolve(result))(type.decode(value))
  );
};

Our decodePromise function handles any input data based on a defined decoder and then returns a promise, based on running the actual decoding operation.

const loadUser = (id: number) => {
  fetch(`http://www.your-defined-endpoint.com/users/${id}`)
    .then((response) => response.json())
    .then((user) => decodePromise(User, user))
    .then((user: User) => state.saveUser(user))
    .catch((error) => {
      console.log({ error });
    });
};

There are more improvements we could make, but we should have a basic understanding of why it might be useful to validate any external data at runtime. io-ts offers more features handling recursive and optional types. Furthermore there are libraries like io-ts-promise that provide more features and useful helpers, the above decodePromise, for example, is available in a more advanced variant via io-ts-promise.

Links

io-ts

io-ts-promise

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Notes on Advanced TypeScript: Transforming Types

A. Sharif — Thu, 28 Apr 2022 14:31:01 +0000

Introduction

Transforming Types

There are situations where you have a defined type, but need to adapt some properties to work for a specific use case. Let's take the following example, where we have defined a Box type:

type Vec2 = { x: number; y: number };
type Box = {
    id: string;
    size: Vec2;
    location: Vec2;
    content: string;
    color: string;
};

This Box type works well, except that we have a user interface, that allows the user to define the size, content, color and even location. The id property might not be defined yet, which prevents us from using the type as is. We need a way to tell our input, that the provided values are a Box with the id property being optional.
The following example will not work:

const defineBox = (box: Box) => {
   // some processing happening here
};
defineBox({
  content: "Content goes here",
  color: "green",
  location: {x: 100, y: 100},
  size: {x: 50, y: 50}
}); 
/** 
 * Fail: Property 'id' is missing in type 
 * '{ content: string; color: string; location: { x: number; 
 * . y: number; }; size: { x: number; y: number; }; }' 
 * but required in type 'Box'. 
 */

TypeScript will complain that the property id is required in type Box. The good news ist that we can transform our Box type to work via defining our own MakeOptional type. By leveraging the built-in types Pick and Omit we can create a type that accepts a defined of keys that we can be converted to optional:

type MakeOptional<Type, Keys extends keyof Type> = 
  Omit<Type, Keys> & Pick<Partial<Type>, Keys>;

Let's take a closer look at what is happening. First we use Omit to remove any keys from the original type and then we make our type partial via the Partial type and pick the previously excluded keys. By joining the two type operations, we can now use the newly created MakeOptional in our previous example.

type BoxIdOptional = MakeOptional<Box, "id">;
const defineBox = (box: BoxIdOptional) => {

};
defineBox({
  content: "Content goes here",
  color: "green",
  location: {x: 100, y: 100},
  size: {x: 50, y: 50}
});

Our defineBox function works as expected now, no matter if the id is provided or not. This good already, but we can do even more type transformations as needed. Let's look at a couple of more scenarios.
We might want to convert all properties by type, for example we would like to convert all properties of type string to number. This can be achieved by defining our own ConvertTypeTo type:

type ConvertTypeTo<Type, From, To> = {
  [Key in keyof Type]: Required<Type>[Key] extends From ? To : Type[Key];
};

By going through all the keys, we check if a key extends the From generic type and convert it to the defined To type.

/**
 * type BoxStringToNumber = {
 *   id: number;
 *   size: Vec2;
 *   location: Vec2;
 *   content: number;
 *   color: number;
 * }
 */
type BoxStringToNumber = ConvertTypeTo<Box, string, number>;

By using the ConvertTypeTo type, we converted all properties of type string to number.
Another scenario might be that we want to include or exclude properties by type. Here we can write a building block type that can extract property keys based on a type.

type FilterByType<Type, ConvertibleType> = {
  [Key in keyof Required<Type>]: Required<Type>[Key] extends ConvertibleType ? Key : never;
}[keyof Type];

Again, we iterate over all the keys for a given type and check if the key extends the type we want to filter on. Any key that does not extend the convertibleType is filtered out by returning never.
A short FilterByType test using our previously defined Box shows that we can retrieve all keys of type string.

// type BoxFilteredByTypeString = "id" | "content" | "color"
type BoxFilteredByTypeString = FilterByType<Box, string>;

Now that we have our FilterByType in place we can write a custom type that either includes or excludes properties by type. To exclude we can use Omit again and combine it with our custom type:

type MakeExcludeByType<Type, ConvertibleType> = 
  Omit<Type, FilterByType<Type, ConvertibleType>>;

To include all properties by type, we only need to replace Omit with Pick:

type MakeIncludeByType<Type, ConvertibleType> = 
  Pick<Type, FilterByType<Type, ConvertibleType>>;

Here is an example showing how we can convert the Box type, by including or excluding all properties of type string.

/**
  type BoxOnlyVec2 = {
    size: Vec2;
    location: Vec2;
  }
 */
type BoxOnlyVec2 = MakeExcludeByType<Box, string>;

/**
  type BoxOnlyNumber = {
    id: string;
    content: string;
    color: string;
  }
 */
type BoxOnlyNumber = MakeIncludeByType<Box, string>;

There are more transformations we can do, like for example making properties required, optional or read only based on a type or types. Here are more examples you can checkout

We should have a basic ideas of how to transform types now.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

An Introduction Into Lenses In JavaScript

A. Sharif — Wed, 12 May 2021 20:00:35 +0000

An Introduction Into Lenses In JavaScript

Functional Getter/Setter

Originally published on Nov 17, 2016 and also available here.

Introduction

If you know the ins and outs of lenses, including when and how to use them, then this walkthrough might not be for you. For everyone else, who might or might not have heard about lenses, this write-up is intended as an introductory into the concept.

Basics

What are lenses? In short, they are functional getter/setter. A short hand functionality for accessing as well as updating data objects. Now what does this look like in the real world and why should we use it? Let’s start off with a simplified example.

const user = {id: 1, name: 'userA'}

To access the user id we would write something like the following.

user.id // 1

Updating could be accomplished like this f.e.

user.id = 2

Now, mutating the user object might not be the best idea, as it might lead to some unexpected behaviours later on. We might assume some value to be defined, but have no guarantees.
So let’s bring in a couple of utility functions that could improve the previous approach.

Getting Started

We’ll add a couple of Ramda functions (you can also use lodash/fp).

assoc: for overriding a specified property and getting a new object in return.
prop: for accessing an object property.
So a cleaner variant of accessing and updating the user id can be achieved by applying these functions.

prop('id', user) // 1
assoc('id', 2, user) // {id: 2, name: 'userA'}

Updating the user id in the second example doesn’t mutate our user object. This is what we want to ensure in the first place.

Due to the fact that we have prop and assoc for updating and retrieving an object’s specified properties, we can start thinking about what lenses mean in this context. We know that lenses are functional getter/setter and our previous examples already enabled us to get and set properties, so let’s write some pseudo code to combine these things together.

const idLens = lens(prop('id'), assoc('id'))
view(idLens, user) // 1
set(idLens, 2, user) // // {id: 2, name: 'userA'}

We introduced a couple of new functions here, so let’s go through them one by one and see how this all fits together.

The first function lens expects two arguments, the first being a getter and the second being a setter. This doesn’t really need too much explanation, contrary to the next two lines. What do view and set do? view expects a lens and an object to apply the lens on. set expects three arguments, the lens the new value and the user. It then updates the defined field by applying the lens with the new value, just like the name implies.

What we have up until now is pseudo code, so to get a better feel, let’s create a naive implementation before we fallback to a tested and ready-to-use solution.

const lens = (getter, setter) => {
  return ({
    get: obj => getter(obj),
    set: (val, obj) => setter(val, obj),
  })
}

Admitted, it’s not the nicest of all solutions, but it should work. Now that we have our own lens function in place, let’s figure out how view and set might work.

const view = (lens, obj) => {
  return lens.get(obj)
}
const set = (lens, val, obj) => {
  return lens.set(val, obj)
}

Actually, we could just call get on the lens object when using view and the lens set method when applying the standalone set function. Rerunning our previous example should return the expected outcome.

const idLens = lens(prop('id'), assoc('id'))
view(idLens, user) // 1
set(idLens, 2, user) // // {id: 2, name: 'userA'}

From here on out let’s neglect our naive implementation and use Ramda instead. Ramda offers a number of very useful lens functions. The following examples rely on lens, lensProp, lensPath, lensIndex, view, set, over and common Ramda functions like compose and map. Now that we have the low level basics covered, let’s see lenses in action. The following examples will be based on the following user object.

const user = {
  id: 1,
  name: 'userA',
  company: {
    id: 12,
    name: 'bar',
    address: {
      street: 'randomstreet',
    }
  },
  comments: [
    {id: 2, text: 'yes, this could work.', to: {id: 4}},
    {id: 3, text: 'not sure.', to: {id: 12}},
    {id: 4, text: 'well, maybe', to: {id: 4}},
  ],
}

Our previous code can be rewritten using the lensProp shorthand function, which returns a lens for getting and setting a defined field. To reiterate on our previous example.

const idLens = lensProp('id')
view(idLens, user) // 1
set(idLens, 2, user) // user = {id: 2, name: 'userA', ...}

Let’s see how we can update nested properties, by retrieving the companies street address. Ramda’s lensPath comes handy in this specific case.

const userCoStrLens = lensPath(['company', 'address', 'street'])
view(userCoStrLens, user) // randomstreet

Updating the street name can be achieved as easily, without mutating the user object.

const updateUserAddress = set(userCoStrLens, 'Teststreet', user)

Updating array values

We can even update specific array values by using lensIndex.

const firstItem = lensIndex(0)

Same can be achieved with lensPath, which can handle keys as well indices.

const firstCommentLensId = lensPath(['comments', 0, 'id'])
view(firstCommentLensId, user) // 2

The firstCommentLensId can also be applied to update that comment id using the set function.

set(firstCommentLensId, 12, user)

Using over to apply a function

We have seen view and set in action, but we haven’t touched a third interesting function called over. With over we can apply a function to update the field of an object or array. Imagine we wanted to uppercase the first comment.

const firstCommentTextLens = lensPath(['comments', 0, 'text'])
over(firstCommentTextLens, toUpper, user)

Composition

Another nice and interesting fact is that lenses compose.

const addressLens = lensProp('address')
const streetLens = lensProp('street')
const addressStreetLens = compose(addressLens, streetLens)

A noteworthy aspect is that they compose from left to right. We can also mix and match lensIndex and lensProp just like in the following example.

const commentLens = lensProp('comments')
const firstIndexLens = lensIndex(0)
const idLens = lensProp('id')
compose(commentLens, firstIndexLens, idLens)

By using map we can even convert all comments to uppercase.

over(commentLens, map(over(textLens, toUpper)), user)

Real World

You still might be asking yourself if this is worth all the trouble, when one can simply update or access an object directly. One use case that comes to mind is that we can pass a lens function around, enabling to retrieve values from a state object without having to know about how this object is actually structured. Another is that we never directly mutate our object or array but get a shallow copy in return.

Lenses should be used when we need to update or extend an object without wanting to break other implementations or where we don’t have access to libraries like immutable.js f.e.

Using lenses when rendering a view for example, where you need to format the given data, is one good example.

const getComments = view(lensProp('comments'))
const getText = view(textLens)
const textToUpper = over(textLens, toUpper)
const allTextToUpper =
  compose(map(compose(getText, textToUpper)), getComments)

Now we can call allTextToUpper which ensures that all are comments are in capital letters minus mutating our original user object.

const renderView = user => (
  <div id="comments">
    {map(comment => (<div>{comment}</div>), allTextToUpper(user))}
  </div>
)

Outro

We should have covered the basics with this write-up.
If you want to read more about lenses and see further examples, I would recommend reading Lenses with Immutable.js by Brian Lonsdorf and Lenses and Virtual DOM Support Open Closed by Hardy Jones.

If you have any feedback please leave a comment here or on Twitter.

Notes on TypeScript: Building a validation library

A. Sharif — Thu, 27 Feb 2020 15:33:46 +0000

Introduction

These notes should help in better understanding TypeScript and might be helpful when needing to lookup up how to leverage TypeScript in a specific situation. All examples in this post are based on TypeScript 3.7.2.

Basics

When building applications, we often have to deal with some user provided input. A common way is to display a form, so that the user can input the data. This data then has to be validated and some feedback should be provided, incase the data is invalid. To achieve this, we validate the data and then display feedback like an error message or multiple messages.

In this post, we will write a small validation library and leverage TypeScript to improve the developer experience.

Our validation library should be framework independent and only take care of validating data, leaving the actual UI representation in user land.

Let's assume that we are provided with some user data object. This could be similar to the following example.

const fieldValues = {
  name: "Test User",
  level: 10,
  description: "Test Description"
};

There is a specification that we need to implement, which specifies that name should not be empty and description should have a minimum length of 10 characters.

What we also want to do is return a validation result object, which we can pass to a UI function and render the error messages incase the data has invalid fields. This might look something like the next example.

const result = {
  name: true,
  level: true,
  description: "Minimum of 10 characters required!"
};

The result shape might differ from case to case. Our validation library should have no assumptions on how the result is structured, except that we're returning an object. In this specific case we want a function that enables us to pass in a collection of validation rules and the previously defined fieldValues and get back a validation result containing either a true, when valid, or an error message string.

const result = validate(validationRules, fieldValues);

Now that we have a general idea of how our library should function from an API perspective, the next step is to implement that library and provide the needed functionality.

Implementation

Before we start implementing the validation library, let's recap on the form field values, which we defined as being represented as an object.
Our validate function should return all or a subset of the provided keys with their corresponding validation result. For our basic implementation, we assume that our field validation functions either return a true or an error message string.

type ValidationResult<T, U> = Partial<{ [Key in keyof T]: U }>;

The ValidationResult type is what our validate function will return. What's still missing is how the validation rules should be defined. To keep as much as possible in user land, our validate function accepts a list of rules. These rules expect the complete field object and then return a validation result containing the complete object or only a subset of it.
This is how we will define a rule.

type Validation<T, U> = (fields: T) => ValidationResult<T, U>;

Let's write one or two validation functions next.

const hasLength = <T>(len: number, input: string | Array<T>) =>
  input.length >= len;

Our hasLength function expects some string or an array and then checks if the provided input is larger or equals the provided minimum length. This newly created predicate function can be a basic building for writing some validation functions.

const hasUserName = (input: string) =>
  hasLength(1, input) ? true : "Name is required.";
const hasValidDescription = (input: string) =>
  hasLength(10, input)
    ? true
    : "Description requires a minimum of 10 characters.";

Next we could define a collection of validation functions to run against some provided input.

const fieldValues = {
  name: "Test User",
  level: 10,
  description: "Test Description"
};

type FieldValues = typeof fieldValues;

/*
type FieldValues = {
    name: string;
    level: number;
    description: string;
}
*/

Our validate library should be able to accept and handle a collection of rules and some input and run these validations against the provided input and return a user land defined result shape.

const validationRules = [
  ({ name }: FieldValues) => ({
    name: hasUserName(name)
  }),
  ({ description }: FieldValues) => ({
    description: hasValidDescription(description)
  })
];

Now that we defined our validation rules we want to run them against our validate function. Technically we want to iterate over all the predicate functions, collect the results and then merge them together into an object, as defined via the ValidationResult type. To recall this is how we defined this:

type ValidationResult<T, U> = Partial<{ [Key in keyof T]: U }>;

Our implementation can be a combination of map and reduce, where we map over the predicates and then merge them into an object.

const validate = <T, U = boolean | string>(
  validations: Validation<T, U>[],
  fields: T
): ValidationResult<T, U> =>
  validations
    .map(validation => validation(fields))
    .reduce((acc, a) => Object.assign(acc, a), {});

We could simplify the code a little more, by running the predicate function inside the reduce as well.

const validate = <T, U = boolean | string>(
  validations: Validation<T, U>[],
  fields: T
): ValidationResult<T, U> =>
  validations.reduce(
    (acc, validation) => Object.assign(acc, validation(fields)),
    {}
  );

Finally we can test this out with some data.

const fieldValues = {
  name: "Test User",
  level: 10,
  description: "Test Description"
};
type FieldValues = typeof fieldValues;

const validationRules = [
  ({ name }: FieldValues) => ({
    name: hasUserName(name)
  }),
  ({ description }: FieldValues) => ({
    description: hasValidDescription(description)
  })
];

validate(validationRules, fieldValues); // {name: true, description: true}

In the above example both fields are valid, now let's test this with some invalid data and check the result.

const fieldValues = {
  name: "Test User",
  level: 10,
  description: "Test"
};

validate(validationRules, fieldValues);

/*
  {
    description: "Description requires a minimum of 10 characters.",
    name: true
  };
*/

The provided description didn't match the expected rule and our validate functionality correctly returned the pre-defined error message.

After going through this short lesson, we should have a good idea how to leverage TypeScript when building small libraries, especially when thinking about the shape of the in/out data.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Notes on Product Development: What Do We Need To Solve This Problem?

A. Sharif — Fri, 21 Feb 2020 13:44:14 +0000

Introduction

When building a product or system there is always another feature to implement. Sometimes it’s very specific in form of a task, describing the required steps in detail, sometimes it’s in form of a story or an epic and might be very vague.
Very often we’re already presented with a solution to a problem. This leaves a sole developer or development team with figuring out the technical details in best case, only implementing the technical solution at worst.

But what if we were able to talk about the problem not the solution?

Let’s try to get a better understanding via an example.

Defining The Feature

The development team has the following feature in the pipeline:

As a business analyst I want to be able to export my data,
so we can process the data inside Excel.

This feature is written as a user story, but might come in different shapes and forms. It already provides us with a solution: "export the data". So we need to export the data as a CSV file f.e.
At this point we’re already focused on figuring out how to get the data out of the system. We already have a solution and we need to figure out how to implement this feature on a technical level.

But what is the actual problem we're trying to solve?

The story card doesn’t provide any further information on what happens once the data is exported.

* Is there a specific calculation that needs to be run outside the system?
* Could the solution be provided by the existing product, instead of having to export the data?

These are just hypothetical questions, but once we’re provided with the solution it becomes harder to step back and understand the problem.

Let’s take the original feature and rewrite it:

As a business analyst we will need to calculate the median and mean 
of all recruiting costs on a per month and department basis 
so we can control the costs during the year.

Even if this story might sound a little exaggerated, this formulation provides us with a very specific problem now.

* Now we might even take another step back and ask what happens with these costs?
* Do they need to be exported as a PDF or CSV?
* Are the results imported back into the existing system?

We can keep stepping back and ask further questions, which might help us to better understand the actual underlying problem.
The outcome might be that we will solve the actual problem differently, instead of being presented with a solution to part of the problem.

Thinking Beyond Features

Now you might be trying to argue that this is a process or methodology problem, that the stories need to be done in a way, that leaves room for asking questions. Take a look at at an existing backlog and see how theses stories are formulated and if they help to step out of a specific feature and enable to understand the bigger picture.

Mostly there is room to discuss the acceptance criteria within a feature, as the solution is often already provided directly or indirectly. What mostly happens is that we shift from thinking about the overall outcome to number of features.
The number of features becomes the driving factor, not the underlying problems these features are set out to solve. Often there is a product owner or manager or a product team tasked with ensuring that there is backlog with prioritized features. This also means that the development team is involved later in the process.

We already have concepts for defining the larger picture, think in terms of epics or even themes.
Themes or epics are vague and help to bound features together, to provide context for these features.

When we want to think about solving problems, this level of abstraction is the interesting part. But very often the solution is broken into multiple features in form of stories or a different format, and discussed in sprint planing or another development team focused meeting.
We’re interested in an abstraction level that helps us to focus on the problem.

Talking About The Problem

What do we gain from talking about the problem?

There are a number of benefits that we can derive from this approach. It enables a team to have the opportunity to provide valuable feedback early on. It enables everyone involved, especially the team doing the actual work, to get a better understanding of the bigger picture.

What is the problem we are trying to solve and why do we need to solve it?

It creates a platform where we can learn and understand the problem first. Now we can also ask the following question:

What Do We Need To Solve This Problem?

This is different from “Here is the solution, now solve it”. Instead of handing-off a problem from A to B, there is a groundwork for discussing what is actually needed to solve the actual problem.

* It helps to clarify the problem not the solution.
* It helps to derive a solution.
* It opens up room for collaboration.
* It opens up a dialogue.
* It’s not simply outsourcing the problem.
* Is this problem even solvable at given time?
* Is there a different problem that needs to be solved first?
* Can we break up the problem in multiple parts?
* Are the right people involved in solving this problem?

Summary

Very often we’re focused on metrics like number of features or development speed and other quantitive metrics.
But sometimes it makes sense to think about what we are actually trying to solve.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Learning TypeScript

A. Sharif — Wed, 12 Feb 2020 19:00:30 +0000

I have written a 16 part "Notes on TypeScript" series, which is an unstructured collection of TypeScript related notes. Trying to convert this series into a more structured learning resource. Here are some questions, where I would be very interested in some feedback.

What do you find hard about TypeScript and what would you like to learn more about?
How could a good learning experience look like?

Notes on TypeScript: Type Level Programming Part 3

A. Sharif — Tue, 11 Feb 2020 18:37:37 +0000

Introduction

TL;DR:

Liquid error: internal

Type Level Programming

This is the third part, where we focus on type level programming in TypeScript. For a better understanding of this post, it might be helpful to have a basic understanding of mapped and lookup types as well as conditional types. Additionally reading the first two posts about type level programming can be helpful too.
You can find the following posts here:

Examples

In this part, we're going to write some helper functions that enables us to do type transformations based on a specified type. There are cases where we might want to make all properties of a specific type readonly or optional or required. There also might be siutations where we need to transform all properties of a defined type to another type.

Let's assume we have User type.

type User = {
  name: string;
  points: number;
  active: boolean;
};

How can convert anything of type boolean to true for example, as there might be a specific part of our application, where a User has to be in an active state? There might be situations where we need to transform all properties of a specific type to a different type. One example would be converting a boolean flag to a string after validating some object and wanting to display error messages.

type User = {
  points: number;
  name: string;
  active: boolean;
};

/**
  type ActiveUser = {
    points: number;
    name: string;
    active: true;
  }
 */
type ActiveUser = ConvertTo<User, boolean, true>;

Our ConvertTo helper should be able to receive some type T and the type we want to convert from as well as the type we want to convert to.

type ConvertTo<Type, From, To> = {
  [Key in keyof Type]: Required<Type>[Key] extends From ? To : Type[Key];
};

The ConvertTo helper maps over the provided type and then checks if a property extends the type we want to transform, either keeping the original type or applying the type we want to transform to. One important note here, is that we need to make the lookup strict via Required<Type>[Key], otherwise any optional properties will be skipped in the mapping function.

Next, we will write some helper functions, which we can use a basic building blocks for writing more specific helpers later on. One basic functionality is to be able to filter based on a defined type. The general use case might not be interesting from a user land perspective, as there might not be too many situations where want to remove all properties that are not of a specific type. The FilterByType function only returns keys that match the type we want to filter by.

type FilterByType<T, U> = {
  [Key in keyof Required<T>]: Required<T>[Key] extends U ? Key : never;
}[keyof T];

Using the FilterByType:

type User = {
  id: number;
  name: string;
  points: number;
  active: boolean;
};

type FilteredUserKeys = FilterByType<User, number>;
// type FilteredUserKeys = "id" | "points"

From the above example we can see that this helper functions filters all the keys that have the type number for the provided User shape. If we recall, Omit and Pick both expect a type and the keys to be applied. By combing Omit or Pick with our previously defined FilterByType helper, we can create two new useful helpers: IncludeByType and ExcludeByType. The next example should help us understand how this works.

type ExcludeByType<T, U> = Omit<T, FilterByType<T, U>>;

type IncludeByType<T, U> = Pick<T, FilterByType<T, U>>;

The two generic helpers ExcludeByType and IncludeByType expect a type T and the type we want to filter by.

type User = {
  id: number;
  name: string;
  points: number;
  active: boolean;
};

/**
  type UserWithoutTypeNumber = {
    name: string;
    active: boolean;
  }
 */
type UserWithoutTypeNumber = ExcludeByType<User, number>;

/**
  type UserWithOnlyTypeNumber = {
    id: number;
    points: number;
  }
 */
type UserWithOnlyTypeNumber = IncludeByType<User, number>;

The above examples are equivalent to writing:

type UserWithoutTypeNumber = Omit<User, 'name' | 'string'>;

type UserWithOnlyTypeNumber = Pick<User, 'id' | 'number'>;

So IncludeByType and ExcludeByType can be used as further building blocks to construct helpers that take care of transforming types to readonly, optional or required. Now that we have all the low level helper functions in place we can combine these helpers with the TypeScript provided helpers. To make a specified properties read-only by type, we can use Readonly and combine it with the type T we want to transform.

type ReadOnlyByType<T, U> = T & Readonly<IncludeByType<T, U>>;

The above code could also be rewritten, to make what is happening more clear, like so:

type ReadOnlyByType<Type, FilterType> = Type & Readonly<IncludeByType<Type, FilterType>>;

Using ReadOnlyByType would make a defined type readonly by filtering all properties that extend the filter-by type, making them readonly and then combining them with original structure.

type User = {
  id: number;
  name: string;
  points: number;
  active: boolean;
};

/**
  type UserNumberReadOnly = {
    readonly id: number;
    name: string;
    readonly points: number;
    active: boolean;
  };
 */
type UserNumberReadOnly = ReadOnlyByType<User, number>;

This enables us to prevent id and points to be overridden in this case.

const user: UserNumberReadOnly = {
  id: 1,
  name: "test",
  points: 20,
  active: true
};

user.id = 3;
// Cannot assign to 'id' because it is a read-only property.
user.points = 30;
// Cannot assign to 'points' because it is a read-only property.

Finally, we might need the capability to transform some type to optional or required for a given shape T. Again, by using the TypeScript provided Partial we can now provide two helper functions that can either set a specific type to optional and all others to required or vice versa. Using our previously defined ExcludeByType or IncludeByType, we can make a specified type U to optional or required.

type OptionalByType<T, U> = Partial<IncludeByType<T, U>> & ExcludeByType<T, U>;

type RequiredByType<T, U> = Partial<ExcludeByType<T, U>> & IncludeByType<T, U>;

This enables us now to transform our User type:

/**
  type UserNumbersOptional = {
    id?: number;
    name: string;
    points?: number;
    active: boolean;
  };
 */
type UserNumbersOptional = OptionalByType<User, number>;

/**
  type UserOnlyNumbersRequired = {
    id: number;
    name?: string;
    points: number;
    active?: boolean;
  };
 */
type UserNumbersRequired = RequiredByType<User, number>;

There might be one last case, where we would like to only convert all properties of defined type to required, but not change the optional/required for any other properties. In this case we can use the TypeScript helper Required.

type OnlyMakeTypeRequired<T, U> = ExcludeByType<T, U> &
  Required<IncludeByType<T, U>>;

OnlyMakeTypeRequired enables us to only make these properties required that fit a defined type U. Our User shape might contain optional and required fields, but only the types that match U will be transformed to required, leaving all other properties as is. Check the following example for better understanding.

type User = {
  id: number;
  points?: number;
  name: string;
  active?: boolean;
};

/**
  type User = {
    id: number;
    name: string;
    points: number;
    active?: boolean;
  }
 */
type RequiredByTypeOnlyUser = OnlyMakeTypeRequired<User, number>;

const user: RequiredByTypeOnlyUser = {
  id: 1,
  name: "Test"
};
// Error! points is required now!
// Property 'points' is missing in type '{ id: number; name: string; }' but required in type 'Required<Pick<User, "id" | "points">>'

As we have seen in the previous examples, we can build helper functions when needed for specific situations by leverage type level programming. If you're interested to learn more about type level programming, you can study the TypeScript provided helpers like Partial, Pick or Readonly, furthermore the first two posts on type level programming (see links below) in the "Notes on TypeScript" series might be helpful as well.

Links

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Notes on TypeScript: Handling Side-Effects

A. Sharif — Tue, 28 Jan 2020 17:04:49 +0000

Introduction

Basics

When working with TypeScript, we want to rely on the defined types throughout the application.
But can we always rely on these types? There are cases where we can't guarantee that these types are valid at compile time. Let's think about some possible cases where this might apply.
When working with external files that we might load at runtime, or a simple API fetch. We can't guarantee that what we expect is actually the expected type at runtime. There might be cases where we control the complete application cycle and can assume that the expected types apply, but even then, we're dealing with an side-effects, that we want to handle safely as possible.

Let's see an example where we assume a specific type is actually valid, but where we might run into problems at runtime.

type Theme = "basic" | "advanced";

type User = {
  id: number;
  name: string;
  points: number;
  active: boolean;
  settings: {
    configuration: {
      theme: Theme;
    };
  };
};

User type defines a specific user shape, nothing too special so far. In our application we will probably be retrieving this user object via an endpoint and use this user object to later update some properties.

Somewhere in our app, we want to enable to change the user settings, to make the profile configurable. changeTheme is a simplified function that accepts a possible theme and updates the user settings.

const changeTheme = (user: User, theme: Theme): User => {
  return {
    ...user,
    settings: {
      ...user.settings,
      configuration: { ...user.settings.configuration, theme }
    }
  };
};

The user object has to come from somewhere, so let's assume we have a function that takes care of loading that user.

const loadUser = () => {
  fetch("http://localhost:3000/users/1")
    .then(response => response.json())
    .then((user: User) => state.saveUser(user))
    .catch(error => {
      console.log({ error });
    });
};

Furthermore we might want to keep state of that user inside the client, so we call the saveUser function to keep track of the object.

type State = {
  user: User | undefined;
  saveUser: (user: User) => void;
  getUser: () => User | undefined;
};

const state: State = {
  user: undefined,
  saveUser: (user: User) => {
    state.user = user;
  },
  getUser: () => {
    return state.user;
  }
};

TypeScript will not complain. Calling the json function on the response object returns an Promise<any>, so there is no guarantee that we are dealing with a User type. The following code will compile as well. If you recall changeTheme expects a User type.

const user = state.getUser();
if (user) {
  const result = changeTheme(user, "advanced");
}

When calling changeTheme with the following fetch result, we will be greeted with an error at runtime. We expected the property to have the settings key, but it actually returned setting.


const user = {
  id: 1,
  name: "Test User",
  points: 100,
  active: true,
  setting: {
    configuration: {
      theme: "basic"
    }
  }
};

changeTheme(user, "advanced");

// Error! 
// Uncaught TypeError: Cannot read property 'configuration' of undefined at changeTheme

This example might be very simplified, but it should show that we can't fully rely on the data coming from an external source. We could be more defensive when working with user, which we should be doing anyway, but it also means that we can rely on these types even less. Adding a check will prevent the previous error.

if (!user || !user.settings || !user.settings.configuration) {
  return user;
}

We have seen that we need to be more restrictive when working with external data and relying TypeScript defined types, but there might be a more convenient way to handle side-effects when working inside a TypeScript codebase.

Advanced

If you have worked with Elm or ReasonMl before, you might have noticed that there is a way to encode and decode any JSON data. This is very valuable as we want to ensure that JSON data we are getting has the expected type. TypeScript by default doesn't offer any runtime JSON validators, but there are a couple of possible solutions to solving that problem in TS land. We will use one of these libraries to show how we can make our types more reliable.

io-ts is a library written by Giulio Canti that is focused on runtime type validations. We will not get too deep into the details and capabilities of this library, as the official README covers most of the basics. But it's interesting to note, that the library offers a large range of so called codecs, which are runtime representations of specific static types. These codecs can then be composed to build larger type validations.

Codecs enable us to encode and decode in and out data and the built-in decode method that every code exposes, returns an Either type, which represents success (Right) and failure (Left). This enables us to decode any external data and handle success/failure specifically. To get a better understanding let's rebuild our previous example using the io-ts library.

import * as t from "io-ts";

const User = t.type({
  id: t.number,
  name: t.string,
  points: t.number,
  active: t.boolean,
  settings: t.type({
    configuration: t.type({
      theme: t.keyof({
        basic: null,
        advanced: null
      })
    })
  })
});

By combing different codecs like string or number we can construct a User runtime type, that we can use for validating any incoming user data.

type UserType = t.TypeOf<typeof User>;

This gives us the exact same representation we defined in the beginning of this post.

type UserType = {
  id: number;
  name: string;
  points: number;
  active: boolean;
  settings: {
    configuration: {
      theme: 'basic' | 'advanced';
    };
  };
};

To clarify the concept, let assume we have a user object and want to validate if it has the expected shape. io-ts also comes with fp-ts as a peer dependency, which offers useful utility functions like isRight or fold. We can use the isRight function to check if the decoded result is valid.

const userA = {
  id: 1,
  name: "Test User",
  points: 100,
  active: true,
  settings: {
    configuration: {
      theme: "basic"
    }
  }
};

isRight(User.decode(userA)); // true

const userB = {
  id: 1,
  name: "Test User",
  points: "100",
  active: true,
  settings: {
    configuration: {
      theme: "basic"
    }
  }
};

isRight(User.decode(userB)); // false

Now that we have some basic understanding of how io-ts functions, let's revisit the original loadUser function and decode the data before we pass it around inside our application. But before we start refactoring let's take a look at one more useful functionality, that will help us when working with the Either type, that the decode returns. fold enables us to define a success and failure path, check the following example for more clarification:

const validate = fold(
  error => console.log({ error }),
  result =>console.log({ result })
);

// success case
validate(User.decode(userA));

// failure case
validate(User.decode(userB));

Using fold enables us to handle valid or invalid data when calling our fetch functionality. The loadUser function could now be refactored to handle these cases.

const resolveUser = fold(
  (errors: t.Errors) => {
    throw new Error(`${errors.length} errors found!`);
  },
  (user: User) => state.saveUser(user)
);

const loadUser = () => {
  fetch("http://localhost:3000/users/1")
    .then(response => response.json())
    .then(user => resolveUser(User.decode(user)))
    .catch(error => {
      console.log({ error });
    });
};

As we can see from the above code, we might handle any incorrect representation by throwing another error. This prevents the data from being passed around in our application. There are more improvement we can make here. Right now, we're being very specific in how we're handling the user decoding. There might be an opportunity to write a general function that handles any promise based data.

const decodePromise = <I, O>(type: t.Decoder<I, O>, value: I): Promise<O> => {
  return fold<t.Errors, O, Promise<O>>(
    errors => Promise.reject(errors),
    result => Promise.resolve(result)
  )(type.decode(value));
};

Our decodePromise function handles any input data based on a defined decoder and then returns a promise, based on running the actual decoding operation.

const loadUser = () => {
  fetch("http://localhost:3000/users/1")
    .then(response => response.json())
    .then(user => decodePromise(User, user))
    .then((user: User) => state.saveUser(user))
    .catch(error => {
      console.log({ error });
    });
};

There are more improvements we could make, but we should have a basic understanding of why it might be useful to validate any external data at runtime. io-ts offers more features like error reporters or handling recursive and optional types. Furthermore there are libraries like io-ts-promise that provide more features and useful helpers, the above decodePromise, for example, is available in a more advanced variant via io-ts-promise.

Links

io-ts

io-ts-promise

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Notes on TypeScript: Recursive Type Aliases and Immutability

A. Sharif — Fri, 24 Jan 2020 13:56:40 +0000

Introduction

Recursive Types

Prior to the 3.7 release, it was was not possible to simply write:

type Item = [string, number, Item[]];

The compiler would complain with: Type alias 'Item' circularly references itself.. This was suboptimal, as there are enough use cases where recursive types would be useful. On a side note it was possible for a type alias to reference itself via a property:

type Item<T> = {
    value: T;
    reference: Item<T>;
}

There was still a way to achieve this, but it required to fallback to using interface. This is how the same type would have been defined before the 3.7 release:

type Item = [string, number, Items[]];

interface Items extends Item {};

It required developers to switch back and forth between type and interface definitions and was more complicated than necessary. With 3.7 we can write:

type Item = [string, number, Item[]];

This is a useful improvement.

Immutability

Now that we learned about the recursive type aliases, let's create an immutable type that we can use to add more guarantees into our application code.

We might want to define a specific Shape type for example.

type Shape = {
    color: string;
    configuration: {
        height: number;
        width: number;
    }
};

With the release of TypeScript 3.4, const assertions where introduced. This would enable us to make a JavaScript object immutable.

const shape = {
    color: 'green',
    configuration: {
        height: 100,
        width: 100,
    }
} as const;

Which would result in:


const shape: {
    readonly color: "green";
    readonly configuration: {
        readonly height: 100;
        readonly width: 100;
    }
};

as const converts the properties of any object to readonly, which would guarantee that our shape object is immutable.

shape.color = 'blue'; 
// Error!  Cannot assign to 'color' because it is a read-only property.
shape.configuration.height = 101;
// Error! Cannot assign to 'height' because it is a read-only property.

But there are limitations with this approach. What if we had a property containing an array for example?

const numberArray: number[] = [];
const shape = {
    color: 'green',
    attributes: numberArray,
    configuration: {
        height: 100,
        width: 100,
    }
} as const;

// This would work...
shape.attributes.push(1);

There is another limitation, for example when working with functions. See the next example.

const transformShape = (shape: Shape) => {
    shape.configuration.height = 101;
    return shape;
}

// Would work...
transformShape(shape);

We might add the Readonly type to ensure that we can't change any values.

const transformShape = (shape: Readonly<Shape>) => {
    shape.configuration.height = 101;
    return shape;
}

// Would work...
transformShape(shape);

From the above example we can note that Readonly would only help us if we tried to change any top level properties.

const transformShape = (shape: Readonly<Shape>) => {
    shape.color = 'red';
    // Error! Cannot assign to 'color' because it is a read-only property.
    return shape;
}

The Readonly type is defined like the following:

type Readonly<T> = { readonly [P in keyof T]: T[P]; }

The problem is that it doesn't work with deep nested structures.

To work around the issue we can build our own MakeReadOnly type, that should ensure we can't mutate any deeply nested properties inside a function body.

As we learned about recursive type aliases just before, we can now create an immutable type definition.

type MakeReadOnly<Type> = {
  readonly [Key in keyof Type]: MakeReadOnly<Type[Key]>;
};

Now that we have our own type definition let's see how we can apply this as compared to the previous examples.

const shape: MakeReadOnly<Shape> = {
    color: 'green',
    configuration: {
        height: 100,
        width: 100,
    }
};

shape.color = 'blue';
// Error! Cannot assign to 'color' because it is a read-only property.
shape.configuration.height = 101;
// Error! Cannot assign to 'height' because it is a read-only property.

We get the same results like in the previous examples, using const assertions. This might not be adding any additional value. But let's see what we gain by using our newly defined MakeReadOnly type when working with functions.

const transformShape = (shape: MakeReadOnly<Shape>) => {
    shape.color = 'red';
    // Error! Cannot assign to 'color' because it is a read-only property.
    shape.configuration.height = 101;
    //Error! Cannot assign to 'height' because it is a read-only property
    return shape;
}

As we can see from the above example, it's not possible to change the values of any nested properties anymore. This gives us more control on how objects are handled inside function bodies.

In summary there is no concise way to guarantee immutability in TypeScript. Some additional work is needed to ensure that at least in specific parts of the application mutating an object or array is limited.

Links

Official Release Notes on Recursive Type Aliases

Offical Release Notes on const assertions

Marius Schulz: Const Assertions in Literal Expressions in TypeScript

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Notes on Product Development: Shared Understanding

A. Sharif — Wed, 22 Jan 2020 12:45:42 +0000

There is a lot of literature focusing on product development, agility, methodologies, processes, frameworks and many other write-ups on anything related to building products.
This series is just a collection of personal notes, learnings and reflections on product development and everything that revolves around it. Mainly it's a collection of notes that focus on collaborative product work, on iterating and learning.

Some people view product development as a rigid process and others see it as exploratory. For some it can be a collaborative effort, for others it might be an isolated process with clear defined boundaries.

Most products evolve from trying to solve a customer problem. Sometimes the initial features are implemented by an individual or a very small team. But what happens when this small setup evolves into a large team or even teams working on these features?

At some point a different process kicks in, roles start to become more specific, specialists start to join and everything becomes more formal.
While early on there is probably a shared understanding on why certain problems needed solving, it becomes harder to keep that shared understanding over time. People might join without having the general understanding or context that the initial members had.

But isn’t making a shared understanding something of high priority?

Let’s talk about what is meant with „shared understanding“ in this context. Very often we have roles, like Product owner, Developer, QA or UX person etc., all specified with certain responsibilities within the process.

There might be a person that „manages“ a backlog of items and writes down the specifications in form of stories or in another format.

Some people might be tasked with implementing these stories according to a predefined specification. Others might create designs and layouts and user workflows based on these specs and then there are others that ensure that the implementation matches these specification or acceptance criteria.

Looking at the above scenario, is there a shared understanding?
Does everyone involved know why this feature is being implemented?

From the above scenario, the „story“ is the artifact, that should create a shared understanding. It should contain some information on the why.
On the other hand, we can’t really tell if there is a shared understanding just from thinking about that scenario. There might be no shared understanding and the specifications being the only common thread. But there might also be a strong common and shared understanding as the collective had long discussions and maybe a couple of sessions to identify and define the specifications.

What we do see, is that some type of hand-off is taking place though. A story is defined, designs/workflows/mockups are created, these are then handed-off to a different group that does the implementation. Once the implementation is done this is handed-off to the next group that ensures the solution is on par with some predefined standard. There might be more hand-offs involved, like passing the implementation to ops, to run the solution and so forth.

Is there a more effective way to generate a shared understanding though?

Let’s think about some scenarios. In Scrum for example, it’s the development team and the product owner. Two separated entities, that meet at a pre-defined meeting like the sprint planning.

But why is there a built in hand-off from the very start?
This might be due to historical reasons, mapping back to classic project management, which was tasked with managing the people and the project. The more modern version is a product person managing the product, but not being part of the implementation team.

But who should be part of that team?

Before we try to answer that question, maybe it would be interesting to start from the customer and work backwards from there.
A customer might need a solution for editing some profile information. Who could understand the actual problem inside the organization?

Multiple people come to mind. Let’s think about this for a moment. It might be sales people, who have information about recurring potential customer requests, but it also might be customer support, which could identify some problems in the workflow or user experience, or hidden features disguised as bug reports.

It might be some marketing person, analyzing help docs or blog posts, that might touch the topic, that customers need solving. But it also might be a product person, that has collected data and is always talking to customers. It might also be a developer that would write some queries to detect how a specific feature is used. It could be a QA person, but it also could be someone from UX, some designer, literally anyone might be able to provide some insights.

But how do we approach it commonly?

We mostly have a specific role that identifies features, that writes down these features and that prioritizes these features. So the information is mostly tied to a specific role. That role should ensure that the right things are built and that everyone understands what is needed to be done, so that the feature can be rolled out to the customer. So a shared understanding now might be tied to that specific role, and the quality of that understanding varies on how well communicated this is, be it in person or via an artifact.

But what if thought about this from a different perspective?

What if the team consists of a diverse range of people with varying expertise?
Obviously there is a thing called „cross-functional“ teams, or t-shaped professionals, but let’s not focus on this too much for a minute. Let’s think about how a team could create a shared understanding and collaborate based on that understanding to come up with a solution, that might solve the problem more effectively. Instead of having a fixed meeting once per iteration, there might be an ongoing discussion and iterating on the problem. Before jumping into the first solution, there might be room for iterating on possible options, on asking further questions and coming up with a better solution.

A team could consist of anyone, that might be able to contribute to the problem and solution or a specified domain or the work to be done. The earlier everyone is involved in the process, the earlier a shared understanding can be manifested.
But let’s map back to the original customer problem. The customer is not really interested in how the solution is sourced, how the process is organized, if we use story points or write stories in general. What the customer is interested in, is having the problem solved.

Keeping this in mind, we can see that there are some opportunities that come with creating a shared knowledge within a team. A team dialog could create more effective solutions, which could lead to less silos and hand-offs.

Synchronization is underestimated. If the knowledge is spread all over the place, it would require continuous synchronization, to leverage the collective knowledge and skill, furthermore, there might be a lack of context, once knowledge is passed down from one individual to the next. As the context keeps changing, there is a continuous need to translate the context to everyone involved. A side-effect is mostly needing more and more meetings to keep everything synchronized.

So what if we identified the problem and are able to quantify the impact, but also include quality aspects into our thinking from a product perspective? What if we are able to build a team of diverse people that can ask customers questions, that can collaborate with the customer, invite the customer to sessions and so forth?

What if we didn’t care about roles and more about who can contribute to the solution?

Seeing it from this perspective could mean the following: a strong focus on the complete line of work needed, from exploration to delivery and maintenance. This also means making the complete process transparent, including where does an idea come from, why is it moved to exploration phase, how is the decision behind making this a worthwhile investment to implement it, what are the metrics for validating that this feature is actually useful and adding value. It would require
shifting from hand-offs to collectively understanding the problem and working from there.

At the end we’re building things to solve problems, not to show some reports or charts based on isolated metrics that have nothing to do with actual customers.

This requires a different way of thinking about the work, it would require moving from focusing on strict workflows to focusing on the work that actually needs to be done. It would require to ask the right questions: „What benefit does this meeting have?“, „Who can contribute?“ or „What are the metrics for the decision process?“.

If we can map back everything to actual customers, we can ask „How does this make the customer’s life easier or simpler or better?“

It would also enable us to value learning. One constant goal might be to learn more with every exploration. Learn more about our customers, learn about how to identify better ways of solving problems.

If anything, we should try to improve shared understanding. Make the information transparent, enable everyone to contribute in a sensible manner and leverage the collective group knowledge for continuous improvement.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Hidden Knowledge In Software Teams

A. Sharif — Sun, 19 May 2019 14:52:02 +0000

Introduction

If you’ve been working in Software long enough you probably have seen the following two situations:

a. Started working on a new system

b. Joined an already running system

Systems evolve and change over time. Another trait they commonly have, is that they become complex, not necessarily on a code level, but in terms of interactions between parts of the system. How does A behave in relation to B? etc. To be fluent within a large system, you need an understanding of how everything is interconnected.

Very often a single contributor or a small number of people have an understanding of how the system works in its entirety. And as long as a single person is working on that system, this person can be productive, very productive even, moving with full speed. But at some point, more people will be needed to maintain and grow that system, as it can’t be handled efficiently by a single person anymore. It needs to scale from a contributor perspective.

What happens naturally, is that development speed has to be exchanged with on-boarding time. It’s a trade-off, balancing short-term speed with long-term maintainability.

It means that the most productive member, has to invest time on-boarding the ones joining. This on-boarding process comes in different shapes and forms, but commonly achieved with the help of documentation and tests and other ways to describe an expected behaviour.

Common Approaches

In written form, we document architecture, sub-systems, the relation to each other. We might have an explicit documentation for setting up the project, in form of a readme file f.e. etc.
Further more, information can be manifested in automation scripts, in project management tools etc. As we can see, there is a lot of work needed when trying to make any knowledge, relevant to a system, explicit.

In reality these artifacts are often either mostly outdated, available in parts or completely missing. One or two people starting to build a system, don’t necessarily need this information because there is a common understanding of the details, as they have written it themselves.
They know why something behaves in a certain way, what happens when they change a certain part and what is needed to achieve a certain task. Due to the deep understanding of the systems internals they also might be able to move fast, confidently make changes and continuously ship features.

Now what happens when the system requires more people, due to reaching a certain size or complexity?

Scaling the Team

Let’s think about a possible scenario where new people join the original authors. These creators need to on-board the new members.
The new group needs to understand design decisions, they need to figure out how everything is interconnected, they need to find out about boundaries and so and so forth. Due to the lack of historical context, they also need to figure out why certain implementations are built in a certain way, which might be obvious to the original contributors. There is hidden information that needs to be made explicit, step by step. From the perspective of the original author, it might seem that nothing is too complex, that reading the source code etc. should suffice. The more complex a system gets, the harder it is for an outsider to understand these things.

Things get tricky, when the original contributors think that everyone has the same understanding as they do. Mostly this is not the case. How does such thinking surface to the top? Some examples. “That should take 5 minutes to implement”, “Obvious”, “That should be easy” etc. Knowledge is contextual, it depends on an existing implementation, on understanding decisions made in the past etc.

But what happens when a team skips this part of the transition?

What happens when the original authors assume that there is no need for further explanation, no need for a deeper documentation etc.?

What happens when the burden is shifted to the new team members?

Hidden knowledge hinders team performance, people depend on one or two individuals to move forward. Most probably the rest of the team will try to understand certain parts by making small changes in isolation to prevent anything from breaking. They might try to go through available commits or comments to get a better understanding of what might happen when they change A. Will changing A have an effect on B or C? etc.

So, opposed to the original writers, new members will move cautiously, double checking any changes and trying to gather more information along the way. But what can we do to improve the transition phase?

As mentioned, further above, it’s an explicit trade-off between current and future productivity. Time needs to be shifted away from developing on full speed to knowledge transfer across the team. The further ahead knowledge is documented, the less stressful this phase is for anyone involved.

Actions

What are possible actions that can help in this situation? Just think about it from the perspective of someone joining an existing setup.

Readme

A readme that at least has information on how to setup a system f.e. There is none or is it outdated? This is a good opportunity to write or update that readme file. It might seem like nothing too special, at first glance. But how often have you checked if a readme file exists when having an initial look at a repository?

Documentation

Documentation can manifest itself in a number of forms, the most conservative being something like a wiki, where high level decisions are described and where you can look up on architecture as well as other fundamental information. Also, documentation can be found in a design system, in a style-guide, via user stories etc. Depending on what we are building, a combination of multiple tools might be needed to sufficiently transport the required information. Sometimes these infos are partially available, which is a good opportunity to start updating or completing when new people join.

Tests

Tests is another way to describe expected behaviour. They can also provide confidence to new developers, due to signalling when something breaks, as opposed to finding out later on via a review or in production. Partly tests could even fall into the documentation category.

Pair or Mob Programming

Another strong approach to transferring knowledge is actually working together on an existing problem. It can highlight hidden knowledge, it can help quickly resolve questions and it provides confidence to new members when working on the actual feature, due to very short feedback cycles. This is also good opportunity to fill any gaps in regards to documentation, especially information that seems straightforward to the original contributors but not to anyone joining later on.

Work together on Stories

If pair or mob programming is not an option, then work together on a story, meaning break it up into small enough tasks. This is better than having every developer working on a single story.

Understanding Context

This one is probably the most important of all, and might hinder all other approaches: Is there a common understanding that there is hidden knowledge not available to every member working on that product/system? Everyone needs to watch out for phrases like “easy” and “should take 5 minutes to implement” etc.

Being helpful and understanding

This point is a logical conclusion of the previous one. Once there is an acceptance for the fact, that not everyone has the same level of knowledge regarding the system, there should be supportive environment for the newly joined. Supportive can mean, always ready to answer questions, provide help etc. Also means not to assume people are under-performing or “don’t get it”, just because it takes the original authors half the time to solve a problem. At the end it’s about moving from individual high-performers to a performing team.

Ship less

Deliberately choose to ship less when in a transformation phase. A team can’t keep on shipping at the same rate, while on-boarding parts of the team. Committing to less, gives room for the on-boarding phase. This is an important investment for future iterations.

There is more that can be done, but we probably have covered the more common ones, which we can build upon on. Other supportive approaches include scripts for importing-data, setting up local environments etc. All these help to further uncover hidden knowledge.

Summary

We know that if we want to go fast, it’s better to work in solo mode, but if we need long-term maintainability and extensibility it’s better to rely on a team. Sometimes it appears that teams can’t perform, relying on one or two high productive developers doing most of the work. Often the problem is hidden knowledge, meaning that not everyone has the same level of experience with or knowledge of a system.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

DEV Community: A. Sharif

Notes on PostgreSQL: Common Table Expressions

Introduction

Basics

Advanced

Recursive Common Table Expressions

Summary

Notes on Advanced TypeScript: Runtime Validations

Introduction

Basic

Advanced

Links

Notes on Advanced TypeScript: Transforming Types

Introduction

Transforming Types

An Introduction Into Lenses In JavaScript

An Introduction Into Lenses In JavaScript

Functional Getter/Setter

Introduction

Basics

Getting Started

Updating array values

Using over to apply a function

Composition

Real World

Outro

Notes on TypeScript: Building a validation library

Introduction

Basics

Implementation

Notes on Product Development: What Do We Need To Solve This Problem?

Introduction

Defining The Feature

Talking About The Problem

Summary

Learning TypeScript

Notes on TypeScript: Type Level Programming Part 3

Introduction

TL;DR:

Type Level Programming

Examples

Links

Notes on TypeScript: Handling Side-Effects

Introduction

Basics

Advanced

Links

Notes on TypeScript: Recursive Type Aliases and Immutability

Introduction

Recursive Types

Immutability

Links

Notes on Product Development: Shared Understanding

Hidden Knowledge In Software Teams

Introduction

Common Approaches

Scaling the Team

Actions

Readme

Documentation

Tests

Pair or Mob Programming

Work together on Stories

Understanding Context

Being helpful and understanding

Ship less

More

Summary