DEV Community: Vincenzo Chianese

Things I discovered while writing Clojure

Vincenzo Chianese — Mon, 30 Nov 2020 18:34:46 +0000

For some motivations I am not going to go into, I found myself writing a piece of software that would transform a data structure representing a query and transform them in SQL statements, something like Datalog

There have been some items that made me spent more time than I was expecting; I guess these are mostly because I still am not that proficient in Clojure and in the spirit of openness (or more like, I wanted to write something no matter what) here is what I've noticed:

I am still not used to the idea of running the REPL and keep it running forever. I’d run it to test some commands and then close it only to realise I needed it again. Doing this more than 30 times — well it’s probably a 3-4 minutes lost in waiting the REPL to spin up
I’ve paused multiple times on functions that have logical conditions and pondered whether it would’t be better to write a multi-method. Then I would do so and thought that it was not really a great idea, and revert the code back.
I’ve noticed an hard to hold desire, when writing a piece of code, to rewrite it more succinctly because you just remembered that there’s a function in the core library that does exactly what you’re trying to do. I’ve rewrote functions multiple times from using if to use cond and then cond->> and then go back to cond.
Sequence functions on strings do not work in the way I was expecting. They all return a char array, so (concat “hello” “ world”) won’t give me the result I was expecting. Then I figured there’s `clojure.string
Since I am still missing the “muscular” memory for the core functions, I had the clojuredocs website opened all the time, looking up functions, usages and doing small tests before writing the actual code
Apparently there are some situations where you have to please the compiler. The multi method dispatcher appears before — so that the subsequent function can use it; then the singular methods can be defined later. This specific case is a circular dependency so order matters. I wish the compiler would figure that out on its own but I understand that in Clojure the reader is single pass. I just wish I’d have know that earlier :)

Here are the two big ones:

Setting up the cognitec test runner took a while. I was not familiar with it, and although it has a decent documentation it took some time to make it run correctly. In particular I had to figure out the class path convention with filenames, the extra-paths to add in deps.edn, the namespace that has to end with “-test". While I haven’t really been counting the time, I feel I’ve spent somewhat between 25 - 35 minutes on this.
This was the worst. When I finally got used to the idea of keeping the REPL opened forever, it just hit back to me very hard. It appears that the REPL does not reload the dispatching multi method function (it’s cached?), so no matter the changes I was doing, the REPL just would not pick them up. I haven’t been able to understand why for very long time and I ended up disassembling the entire program chasing an ArityException that I simply was not there. I figured exclusively because I closed the REPL one more time by mistake and when reloading, it worked again. Suspicious, I googled the issue and got some confirmations. I do not know how much time I spent on this, but it was significant for sure.

Believe me, when I figured this one out I was undecided whether throwing the computer or myself from the window :)

Succinct JavaScript?

Vincenzo Chianese — Mon, 28 Sep 2020 16:18:10 +0000

It goes with no saying, I am really liking Clojure lately.

Although I do not use in my day to day job (yet?), just in the same way I did with Haskell with regards of Prism, I like to see how other languages approach problems and try to apply their ideas back to TypeScript.

Last week I've been playing with the Game of pure strategy problem, and try to implement a possible solution in Clojure.

After some back and forth and some good feedback from the Clojurians Slack channel, I ended up with a next state function that looks like this:

(def initial-state "Initial game state"
  {:first-player {:deck #{1 2 3 4 5 6 7 8 9 10 11 12 13} :score 0}
   :second-player {:deck #{1 2 3 4 5 6 7 8 9 10 11 12 13} :score 0}
   :bounty-deck #{1 2 3 4 5 6 7 8 9 10 11 12 13}})

(defn random-card-strategy [deck] (rand-nth (seq deck)))
(defn highest-card-strategy [deck] (apply max (seq deck)))
(def draw-card random-card-strategy)

(defn game-step [current-state]
  (let [{:keys [bounty-deck first-player second-player]} current-state
        drawn-card (draw-card bounty-deck)
        first-player-card (random-card-strategy (:deck first-player))
        second-player-card (highest-card-strategy (:deck second-player))
        match-winner (if (> first-player-card second-player-card) :first-player :second-player)]
    (-> current-state
        (update-in [match-winner :score] inc)
        (update-in [:bounty-deck] disj drawn-card)
        (update-in [:first-player :deck] disj first-player-card)
        (update-in [:second-player :deck] disj second-player-card))))

It's kind of known that LISP is generally a very dense language; if we also take in consideration the rich set of Clojure's seq functions and macros, turns out that small chunks of code can do a lot of things in Clojure.

I am (still) primarily a JavaScript/TypeScript developer, and I have been asking "What is the most succinct code that I can write to achieve the same end result?

I cobbled something some code as fast as possible and this is the base line I started from:

const initialState: State = {
  bountyDeck: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13],
  firstPlayer: {
    score: 0,
    deck: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
  },
  secondPlayer: {
    score: 0,
    deck: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
  }
}

const randomCardStrategy = (deck: number[]) => deck[Math.floor(Math.random() * deck.length)];
const highestCardStrategy = (deck: number[]) => Math.max.apply(null, deck);
const drawCard = randomCardStrategy;

const gameStep = (currentState: State): State => {
  const drawnCard = drawCard(currentState.bountyDeck);
  const firstPlayerCard = randomCardStrategy(currentState.firstPlayer.deck);
  const secondPlayerCard = highestCardStrategy(currentState.firstPlayer.deck);
  const matchWinner = firstPlayerCard > secondPlayerCard ? 'firstPlayer' : 'secondPlayer'

  return {
    bountyDeck: currentState.bountyDeck.splice(currentState.bountyDeck.indexOf(drawnCard), 1),
    firstPlayer: {
      deck: currentState.firstPlayer.deck.filter(c => c >== firstPlayerCard),
      score: matchWinner === 'firstPlayer' ? currentState.firstPlayer.score++ : currentState.firstPlayer.score
    },
    secondPlayer: {
      deck: currentState.secondPlayer.deck.filter(c => c >== firstPlayerCard),
      score: matchWinner === 'secondPlayer' ? currentState.firstPlayer.score++ : currentState.firstPlayer.score
    },
  }
}

Note that I have omitted the type definitions in this snippet, since they're not part of the actual code.

It's clear enough that the problem here is that JavaScript does not have a great story to mutate stuff in place, but we can probably do something better by using a Lens library: monocle-ts

Without the boilerplate, the final code that we should take in account is:

const gameStep = (currentState: State): State => {
  const drawnCard = drawCard(currentState.bountyDeck);
  const firstPlayerCard = randomCardStrategy(currentState.firstPlayer.deck);
  const secondPlayerCard = highestCardStrategy(currentState.firstPlayer.deck);
  const matchWinner = firstPlayerCard > secondPlayerCard ? "firstPlayer" : "secondPlayer";

  return pipe(
    currentState,
    matchWinner === "firstPlayer" ? incrementFirstPlayerScore : incrementSecondPlayerScore,
    removeBountyDeckCard(drawnCard),
    removeFirstPlayerDrawnCard(firstPlayerCard),
    removeSecondPlayerDrawnCard(secondPlayerCard)
  );
};

…which is not exactly that far from the Clojure counterpart.

JavaScript is missing a good story for data manipulation out of the box. You need lenses and/or collection libraries to effectively manipulate stuff. Still, it's not an excuse to write bad code.
Without the dot notation (a.b.c), destructuring and threading macro become very important in Clojure. Without them this:

(-> structure :first-player :deck :card :somethingelse)
would be:
(:somethingelse (:card (:deck (:first-player structure))))

which I guess ain't that cool.

P.S: I am aware of ImmutableJS. However their story around typings is not that polished as the one that monocle-ts is offering, so I removed it early from the selection.

Break functions apart

Vincenzo Chianese — Mon, 21 Sep 2020 19:34:47 +0000

For a while now I have been exploring the Clojure programming language. I ended up doing this almost exclusively driven by Rich Hickey's presentations.

Design, composition and performance is among my favorite ones, and there are two specific things that burned in my mind since I saw them:

Design means break things apart in such a way that they can be put back together.

…and

Whenever there is a problem, it's likely because we haven't broken things apart enough.

Since I saw that video, I've kept these in mind every time I would be looking at some piece of code — and it turns out n.2 applies so many times — way more than I was expecting.

This is a small walkthrough of a refactoring opportunity I came across recently while working on a fairly large codebase.

Suppose we have a node type and some functions that grabs a token from somewhere and — in case a specific path is defined, it will need to grab the parent node before continuing the processing:

type AppNode = {
  id: string;
  path: string;
};

declare function extractDataFromNode(n: AppNode): string;

async function getNodeData(
  startNode: AppNode,
  getToken: () => Promise<string>,
  getParentNode: (data: AppNode, token: string) => Promise<AppNode>
) {
  const token = await getToken();

  if (startNode.path === '/p') {
    const parent = await getParentNode(startNode, token);
    return extractDataFromNode(parent);
  }

  return extractDataFromNode(startNode);
}

This function will probably do its job, but frankly speaking, this is a mess in my eyes:

The function name claims to return the node data; on the other hand is also retrieving an authentication token from the system in case is necessary and will happily fallback to the parent node if required.
The function requires two additional function that are used to get the data in case the computation fails for some reason.

When asking/arguing why about the n.2 — the response I'd usually get is that such "pattern" will improve the testability, because I can pass mocks function when testing the code, something along these lines:

describe("returns parent data when path starts with /", () => {
  const tokenMock = jest.fn().mockResolvedValue("token");
  const parentNodeMock = jest.fn().mockResolvedValue({ id: "9", path: "main" });

  const data = getNodeData(
    { id: "0", path: "/s/10" },
    tokenMock,
    parentNodeMock
  );

  return expect(data).resolves.toEqual({}); // assertions
});

I find this not a valid argument; to me this is more a tape-patchy way to fix something that has been put together incorrectly.

Whenever there is a problem, it's likely because we haven't broken things apart enough.

First — let's move the token logic away from this function, and pass it as a regular argument.

async function getNodeData(
  startNode: AppNode,
  token: string,
  getParentNode: (data: AppNode, token: string) => Promise<AppNode>
) {
  if (startNode.path === "/p") {
    const parent = await getParentNode(startNode, token);
    return extractDataFromNode(parent);
  }

  return extractDataFromNode(startNode);
}

By changing the function signature to ask for the token directly instead of taking it on our own, we have effectively moved a non pure operation out of this function. That is a good thing.

At this stage, we might noticed that the token argument is used exclusively to feed the getParentNode function — that we're passing as an argument.

We can do something about it:

- getParentNode: (data: node, token: string) => Promise<node>
+ getParentNode: (token: string) => (data: AppNode) => Promise<AppNode>

Now we can externally bind the token, and get rid of such dependency:

async function getNodeData(
  startNode: node,
  getParentNode: (data: AppNode) => Promise<node>
) {
  if (startNode.path === "/p") {
    const parent = await getParentNode(startNode);
    return extractDataFromNode(parent);
  }

  return extractDataFromNode(startNode);
}

…and then call the token-free function.

const tokenizedGetParentNode = getParentNode("tokenValue");

getNodeData(
  { id: "test", path: "/p/10" },
  tokenizedGetParentNode
);

The code is now inarguably better; still, we're passing the getParentNode — which is not good.

We could reiterate with the same solution we applied above to the getParentNode function, so that we'd be passing the parent node ahead of the time; but it might be a waste of resources: the parent node is only required on certain conditions.

Sometimes sacrificing performances/efficiency for readability is a good call. However for this use case we're going to assume that fetching a parent node is a very expansive operation that we really want to avoid.

Let's break this function apart and reduce its scope to only work with the current node. If the parent node is required, we're going to make this fail and the caller will have to sort this out:

async function getNodeData(startNode: AppNode) {
  if (startNode.path === '/p') {
    return false;
  }

  return extractDataFromNode(startNode);
}

and then the caller can do its job to understand if the parent node call is required or not.

We have now successfully broken the things apart, with all the pieces doing ONE thing. It's now time to put them back together.

declare function getParentNode: (data: AppNode) => RTE.ReaderTaskEither<string, Error, AppNode>;
declare function extractDataFromNode(n: AppNode): string;

function getNodeData(startNode: AppNode) {
  if (startNode.path === "/p") {
    return E.left(new Error("parent node required"));
  }

  return E.right(extractDataFromNode(startNode));
}

const getParentNodeData = (startNode: AppNode) =>
  pipe(getParentNode(startNode), RTE.map(extractDataFromNode))(token);

const node: AppNode = { id: "10", path: "/path" };

return pipe(
  TE.fromEither(getNodeData(node)),
  TE.orElse(() => getParentNodeData(node))
);

Now go, and break your functions apart.

Then break your systems apart.

Then put them back together.

From 0 to Reader

Vincenzo Chianese — Fri, 18 Sep 2020 14:27:38 +0000

The following notes come from an internal discussion I had with some coworkers with no pretension to be an accurate explanation of the Reader monad. Still, my teammates claimed they were helpful to understand the concept; so better put them online.

We'll start with a function whose job is to insert an user in a database:

type User = {
  username: string;
  age: number;
};

declare function createUser(
  user: string,
  details: unknown
): Promise<User>;

Let's write some code to implement the function:

type User = {
  username: string;
  age: number;
};

declare function userExists(user: string): Promise<boolean>;

declare function createUserAccount(
  user: string
): Promise<boolean>;

declare function runAutomaticTrigger(
  user: string
): Promise<boolean>;

async function insertInDb(user: User): Promise<boolean> {
  const db = [];
  db.push(user);

  return runAutomaticTrigger(user.username);
}

async function createUser(details: User): Promise<User> {
  const isPresent = await userExists(details.username);

  if (isPresent) {
    const inserted = await insertInDb(details);

    if (inserted) {
      const accountCreated = await createUserAccount(
        details.username
      );

      if (accountCreated) return details;
      else throw new Error("unable to create user account");
    } else throw new Error("unable to insert user in Db");
  } else {
    throw new Error("user already exists");
  }
}

Now let's say that somebody comes says we need to add logging with this object.

type Logger = {
  info: (msg: string) => undefined,
  debug: (msg: string) => undefined,
  warn: (msg: string) => undefined,
  error: (msg: string) => undefined,
};

Additionally, let's put the constraint in place that the logger is not a singleton instance — thus it's an instance that needs to be carried around.

declare function userExists(user: string, l: Logger): Promise<boolean>;

declare function createUserAccount(user: string, l: Logger): Promise<boolean>;

declare function runAutomaticTrigger(user: string, l: Logger): Promise<boolean>;

async function insertInDb(user: User, l: Logger): Promise<boolean> {
  const db = [];
  db.push(user);

  l.info("User inserted, running trigger");

  return runAutomaticTrigger(user.username, l);
}

async function createUser(details: User): Promise<User> {
  const isPresent = await userExists(details.username, l);

  if (isPresent) {
    const inserted = await insertInDb(details, l);

    if (inserted) {
      const accountCreated = await createUserAccount(details.username, l);

      if (accountCreated) return details;
      else {
        throw new Error("unable to create user account");
      }
    } else {
      throw new Error("unable to insert user in Db");
    }
  } else {
    {
      throw new Error("user already exists");
    }
  }
}

Two things aren't really cool with such approach:

I have to pass the logger in every single function that needs this — every function must be aware of the new dependency
The logger is a dependency, not really a function argument.

To start fixing this, let's try to put the dependency elsewhere:

- declare function userExists(user: string, l: Logger): Promise<boolean>;
+ declare function userExists(user: string): (l: Logger) => Promise<boolean>;

So that we change the way we use the function:

- const promise = userExists(user, logger);
+ const promise = userExists(user)(logger);

The result is:

declare function userExists(user: string): (l: Logger) => Promise<boolean>;

declare function createUserAccount(
  user: string
): (l: Logger) => Promise<boolean>;

declare function runAutomaticTrigger(
  user: string
): (l: Logger) => Promise<boolean>;

function insertInDb(user: User) {
  return (l: Logger) => {
    const db = [];
    db.push(user);

    return runAutomaticTrigger(user.username)(l);
  };
}

async function createUser(details: User) {
  return async (l: Logger) => {
    const isPresent = await userExists(details.username)(l);

    if (isPresent) {
      const inserted = await insertInDb(details)(l);

      if (inserted) {
        const accountCreated = await createUserAccount(details.username)(l);

        if (accountCreated) return details;
        else {
          throw new Error("unable to create user account");
        }
      } else {
        throw new Error("unable to insert user in Db");
      }
    } else {
      {
        throw new Error("user already exists");
      }
    }
  };
}

Let's now introduce a type to help us out to model this:

type Reader<R, A> = (r: R) => A;

And so we can now rewrite userExists as:

- declare function userExists(user: string): (l: Logger) => Promise<boolean>;
+ declare function userExists(user: string): Reader<Logger, Promise<boolean>>;

Since TypeScript does not support HKT (but I still pray everyday that eventually it will), I am going to define a more specific type

interface ReaderPromise<R, A> {
  (r: R): Promise<A>
}

So I can make the following replacement:

- declare function userExists(user: string): Reader<Logger, Promise<boolean>>;
+ declare function userExists(user: string): ReaderPromise<Logger, boolean>;

…and if I define an helper function called chain:

const chain = <R, A, B>(ma: ReaderPromise<R, A>, f: (a: A) => ReaderPromise<R, B>): ReaderPromise<R, B> => (r) =>
  ma(r).then((a) => f(a)(r))

I can now rewrite the entire flow in such way:

function createUser(details: User): ReaderPromise<Logger, User> {
  return chain(userExists(details.username), (isPresent) => {
    if (isPresent) {
      return chain(insertInDb(details), (inserted) => {
        if (inserted) {
          return chain(createUserAccount(details.username), (accountCreated) => {
            if (accountCreated) {
              return (logger) => Promise.resolve(details);
            } else {
              throw new Error("unable to insert user in Db");
            }
          });
        } else {
          throw new Error("unable to create user account");
        }
      });
    } else {
      throw new Error("user already exists");
    }
  });
}

but that ain't that cool, since we're nesting nesting and nesting. We need to move to the next level.

Let's rewrite chain to be curried…

- const chain = <R, A, B>(ma: ReaderPromise<R, A>, f: (a: A) => ReaderPromise<R, B>): ReaderPromise<R, B> => (r) =>
  ma(r).then((a) => f(a)(r))
+ const chain = <R, A, B>(f: (a: A) => ReaderPromise<R, B>) => (ma: ReaderPromise<R, A>):  ReaderPromise<R, B> => (r) =>
  ma(r).then((a) => f(a)(r))

Well what happens now is that I can use ANY implementation of the pipe operator (the one in lodash will do), and write the flow in this way:

function createUser2(details: User): ReaderPromise<Logger, User> {
  return pipe(
    userExists(details.username),
    chain((isPresent) => {
      if (isPresent) return insertInDb(details);
      throw new Error("user already exists");
    }),
    chain((inserted) => {
      if (inserted) return createUserAccount(details.username);
      throw new Error("unable to create user account");
    }),
    chain((accountCreated) => {
      if (accountCreated) return DoSomething;
      throw new Error("unable to create user account");
    })
  );
}

I can introduce another abstraction called Task

type Task<T> = () => Promise<T>

and then, just for commodity

type ReaderTask<R, A> = Reader<R, Task<A>>

Then I can refactor this part a little bit:

- declare function userExists(user: string): Reader<Logger, Promise<boolean>>;
+ declare function userExists(user: string): ReaderTask<Logger, boolean>;

It turns out fp-ts already has a bunch of these defined, so I'm not going to bother using mines:

import * as R from "fp-ts/Reader";
import * as RT from "fp-ts/ReaderTask";
import { pipe } from "fp-ts/pipeable";

type User = {
  username: string;
  age: number;
};

type Logger = {
  info: (msg: string) => void;
  debug: (msg: string) => void;
  warn: (msg: string) => void;
  error: (msg: string) => void;
};

declare function userExists(user: string): RT.ReaderTask<Logger, boolean>;
declare function createUserAccount(
  user: string
): RT.ReaderTask<Logger, boolean>;
declare function runAutomaticTrigger(
  user: string
): RT.ReaderTask<Logger, boolean>;

function insertInDb(user: User): RT.ReaderTask<Logger, boolean> {
  const db = [];
  db.push(user);

  return runAutomaticTrigger(user.username);
}

function createUser(details: User): RT.ReaderTask<Logger, Promise<User>> {
  return pipe(
    RT.ask<Logger>(),
    RT.chain(l => userExists(details.username)),
    RT.chain(isPresent => {
      if (isPresent) {
        return insertInDb(details);
      } else {
        throw new Error("user already exists");
      }
    }),
    RT.chain(inserted => {
      if (inserted) {
        return createUserAccount(details.username);
      } else {
        throw new Error("unable to create user account");
      }
    }),
    RT.map(accountCreated => {
      if (accountCreated) {
        return Promise.resolve(details);
      } else {
        throw new Error("unable to insert user in Db");
      }
    })
  );
}

What are the differences with the original, naive, solution?

Functions are not aware of the dependency at all. You just chain them and inject the dependency once: const user = await createUser(details)(logger)()
The logger is now a separate set of arguments, making really clear what is a dependency and what is a function argument
You can reason about the result of the computation even though you haven't executed anything yet.

Relocating during the COVID-19

Vincenzo Chianese — Thu, 11 Jun 2020 13:58:50 +0000

TL; DR: I had the pleasure of relocating exactly a week or so before the COVID-19 spread to a level where the United States decided to declare the shelter in place and lock down the country.

This is a quick and hopefully humorous tale about what happened to me.

For those of you that do not really know me — I have relocated 3 times over the last 6 years; and by relocation I do not mean moving to a house behind the corner or to a nearby city. I mean changing 3 states and a continent.

Prague, December 2014

I had already decided that it was time for me to leave my original country (Naples, Italy) with the clear idea of not coming back for a very good chunk of time.

At that time, I was so desperate to leave that my desidered standard levels were basically 0. Anything would have done the job. I indeed ended up in Prague, Czech Republic — landing on 12/14/2014.

That ultimately was a very good experience, from both a working perspective (I had a great gig for a local startup) and from personal life as well.

Madrid, September 2017

Once the company I was working for was sold to Oracle in January 2017, I started to talk with my managers about opportunities of leaving the country and work from somewhere else along with other requests that have been on hold for a long time.

I wasn't satisfied with the answers I received — and Oracle was not worth my time in Prague, and decided that leaving would be the n. 1 priority.

I quickly found a gig in Madrid, Spain. It was a place that I always wanted to go and the offer was good, so I agreed.

After one month offline (August), on September 4 I moved.

Austin, March 2020

Time passes, in 2018 I was hired by Stoplight and after they realised I was a good fit, we agreed that moving to the United States would be a good idea.

Unexpectedly, I was selected in the H1B lottery AND was approved by the USCIS at the first shot without questioning or asking additional documentation.

I went from the mood I had at the beginning of the application "we'll try, but most likely we won't make it" to "this was clearly destiny, the United States is calling me".

I got my VISA stamped in December and I decided that January 2020 would be my moving date.

For personal reasons, I couldn't finish all the things I had in mind in January and I shifted the relocation, aiming the beginning of February.

Since I had a speaking engagement at the end of February in Atlanta, I decided that it'd have been a better idea to do a single trip (Madrid - Atlanta) and then move to Austin to settle down.

My time in Europe was about to end and I wanted to use it all. I traveled every weekend of January and February. Me and my flatmate wanted to enjoy our time together till the very last drop and so we went to Granada, Berlin and then Tenerife.

Tenerife was already on the spot because COVID cases were signaled at the time but we all felt safe anyway, since we had very little knowledge about this virus. We were still in the phase of "it’s only in Wuhan")

In any case, I took my plane, I landed in Miami first (where somebody was waiting for me just outside of the plane to bring the for the first inspection — had no problem with that; did my work in Atlanta and then I ultimately landed in Austin.

Then, the COVID and the restriction started.

Flight schedule: gone

I had different flights all booked ahead of the time that I had to cancel. Namely:

Date	From	To
Mar 19	Austin	Osaka
Mar 29	Osaka	Singapore
Apr 4	Singapore	Bali
Apr 7	Bali	Singapore
Apr 7	Singapore	Madrid
Apr 20	Madrid	Amsterdam
Apr 23	Amsterdam	Madrid
Apr 27	Madrid	Austin
May 31	Austin	London
Jun 16	Madrid	Austin
Jul 31	Austin	Madrid
Dec 8	Austin	Madrid

When the whole thing started, the trip in Asia was the first one to be gone. I purchased another flight to reach Madrid anyway in April through a connection in Miami. Such flights ended up being cancelled anyway.

I could have taken the one for June (they can't prevent me because of my passport), but I when I phoned the CBP officer in Dallas to understand what would be the rules to enter the country again they were very clear that if I go out, I would not be able to come back.

Further, once landed in London there would be no flight out for different days; therefore I would be blocked there and it was just not an option.

Current situation: I have spent a significant amount of time on phone calls with different airlines (American Airlines, Scoot, Qatar, Iberia, British Airways). Some of them have refunded me, some of them have offered credit that I decided to keep since I fly with them pretty often.

I still have a significant amount of money out because of this (mostly with American Airlines).

The loop

Once you land in the United States, the first thing you want to do is likely get a Social Security Number, as it's really the door for a lot of other things you probably are going to need such as a bank account, a driving license/ID, set up your health insurance and so on.

When I landed, the offices were already starting to wind down because of the COVID spread, so I ran to the only one still open in Austin and went there 1h ahead of the opening time with no appointment. It goes with no saying that the queue was immense — and with some very weird people there as well.

Once I got that figured out — the employee told me it would take a couple of weeks before I'd get the SSN mailed to my address. That wouldn't normally be a problem, but when also that office closed, I was uncertain if my file had been processed already or not. Without a SSN I was in a situation where:

I couldn't open a bank account
Without a bank account, getting paid would have been way more difficult
I could not enable my health benefits
I could not get a Texas ID
I could not rent a place — at least not through a regular leasing agreement. The only way would have been a sublet, which I wanted to avoid if possible.

I had the luck of being hosted by a very good friend of mine and, although he didn't have any problem hosting me for all the necessary time I wanted to get out of his way as soon as possible. I'm not the kind of person that takes advantage of situations.

The SSN fortunately arrived 8 days later (that was fast, according to what people told me around) and I was able to register for my health benefits. On the other hand, it wasn't really that easy with the rent and the bank account.

The leasing agency tried to verify my credit score, although I told them ahead of the time I had literally landed 10 days ago, resulting in my file being empty. After being refused and some begging, they agreed to rent me the place anyway with a big deposit (3 months) paid in advance. I liked the place very much for a number of reasons, so I agreed.

Regarding the bank account, the experience was even worse. Since all the physical offices were locked down, I went ahead and tried multiple online services. Chime, N26, Varo, Simple, Capital One you name it.

They would all refuse me with no explanation and calling them on the phone wouldn't help. They wouldn't disclose why I was not eligible for a bank account, and my only clue would be insufficient credit history.

While walking on the street I realised the strong presence of the Spanish bank group BBVA. Although I was not their customer when I was living in Madrid — I tried to call them and see if they could help me somehow. Here's a summary of the phone conversation:

Sure no problem, just come here in person and we'll set up a bank account in less than an hour.

Fantastic. When's a good time to come?

Now? Oh now impossible, we're not letting anybody in with the pandemic.

End result: no bank account in any case. I ended up resolving by opening an account with Revolut which had recently opened their operations in the United States; they wouldn't usually do that but since I am a customer in the EU as well, they have "converted" my account to an US one without having to go through the whole procedure of opening a new one.

Revolut has some limitations: it does not support ACH transfers / Direct Debits and does not provide regular checks — which apparently are still important here, but at least I could get paid and (with a 2.3% surplus) I would be able to pay my rent as well.

Building a credit score

I was aware that credit score here is a thing. What I didn't know was how big of a deal it is.

I never had a credit card while in Europe — and I was not planning to have one in the US either; I then realised at my own expenses that credit score is a little bit more than a simple score for merely credit purposes. It tells a lot about you and essentially for anything that is a monthly subscription — people might ask and check your credit score.

My credit score has been checked so many times that I felt that even buying a coffee would require your credit score to be above 650.

From what I've seen so far, there are 3 ways to build your credit score:

Ask for a secured loan. Essentially a company is going to theoretically lend you some money; in reality they won't give you anything but you're still paying the monthly instalment. In exchange for that, the company will report your payments to the credit score agencies. At the end of the repayment, you'll get all your money back minus interests and fees. Fundamentally, it's like asking for a loan and get only its negative parts
Ask your landlord to report your payments
Get a credit card (secured or not) and pay its balance on time

The first one felt so ridiculous that I imperatively refused to do that, so I went for the option n. 2.

Good thing, the community manager of the property I'm living has already an agreement with an automatic platform to report the rent payments. I would just need to register, pay the 6.95$/month fees and I would be good to go.

I did that, and 20 days later I got an email that my application was canceled. I called the support to understand why and the conversation went like this:

My application has been rejected, I would like to understand why.

Sure, let me check that out…oh yeah, we can't accept your application because you do not have any credit score

Ehm, but the purpose of all this is to build a credit score.

Oh I'm so sorry, I understand, but I can't help you. Is there anything else I can help with?

Option 2, gone. I had no choice but to try with a credit card. I tried to apply to the Apple one, and I got rejected. Insufficient credit score.

Ok fair. Then I tried with Discover. Rejected. All right, I'll go with a secured credit card and go from there.

Discover has a good offer, so I decided to go with them. I filled out the form, got the application approved. Unfortunately when it's time to pay for the deposit I realise they only support payments through direct debits, which my bank does not support. As long as I liked Discover, I couldn’t work with them for the time being.

All right, time to find another alternative. I check out on the internet relentlessly and I land on OpenSky CC webpage. They claim to accept the deposit via Debit Card, and it effectively worked.

I filled up the application and I got approved for the pharaonic amount of 100$ per month (Las Vegas, here I come) but you know everybody has to start from something, so I'll go with that.

Once the card finally arrives and I swipe it a couple of times, I realise that they accept a debit card to pay for the deposit, but the only option to pay the monthly due amount is via direct deposit or mail a check. Guess what? My bank does not support any of these.

I needed to find a way to pay back my due amount as soon as possible, otherwise I would start very bad in terms of credit score.

I started to google any single combination of keywords about credit, bank account, checks, credit cards until I found https://www.plastiq.com. The basic idea is simple: you swipe your card, Plastiq will mail a check to the designated recipient.

Although complicated, I thought I finally found a way to close the loop with my secured credit card and slowly build the credit score. I sign up for the service and I issue an order to mail a check to pay my due balance. I went to sleep convinced that I’d have fixed it.

Unfortunately for me, my tale with this wasn’t over yet. I received an email 4-5 days later with a “good” news from Plastiq:

I’m back to the beginning. I can’t pay for my secured credit card balance — and the due date is almost coming. I needed to find a bank account with direct debit support. There was no other option.

I started very early the following morning with a sole purpose: finding somebody willing to open me a bank account online (or even in person, didn’t really matter) with direct debit support.

I’ll skip the whole tale and the 3000 phone calls I had to do. The ultimate choice was between Betterment and Radius and, after a long chat with the support person on the phone, Radius approved to open a bank account right away with full support for paper checks and direct debit. I was finally done. I transferred some money on it as soon as it was opened and paid the secured credit card balance in time.

Totally out of curiosity/boredom I applied again for a Discover Credit Card — not even secured — and it got approved right away for a line of 1000$. I was very very surprised. I guess that the first payment was reported and the fact of having an US Bank Account probably helped during the qualification process.

The skin rashes

I am not allergic to anything.

I’ve been saying this sentence since forever; not because I wanted to brag myself; but rather because it was my belief since my childhood.

How wrong I was — and if you’re thinking about the same, it’s highly probable that you’re wrong too.

Austin is one of the worst places in terms of allergies — and I knew that when I was relocating. I really never worried about that though since by the time I was about to relocate I had visited the city multiple times (October 2018, May 2019, August 2019, November 2019) for a relatively long time (2 to 6 weeks) and I never had any problem.

So after a couple of weeks I’d have moved, welts would start appearing on my body, but since they were few (4-5), localised and weren’t bothering me at all, I didn’t care at all.

The situation degenerated quickly — when I started to have them almost everywhere in my body; I would start to feel itchy; wake up during the night because I would be itchy, and some occasions even painful.

Initially (since I was still convinced that I wasn’t allergic) I thought about bed bugs. I inspected all my furniture (for the record, it was all brand new) and I effectively did not find anything; still, I was still paranoid about it and called the leasing office for an inspection/sanitisation and left the place just to be sure, being hosted by a friend of mine here in Austin for an entire week.

In the meantime, I started to look for a doctor to get a diagnosis and eventually a treatment, if necessary. I looked for a primary physician and a dermatologist.

Because of the pandemic, unfortunately, nobody was willing to see me in person and all they would be able to offer would be video calls, which I refused for some time until I started to see the symptoms worsening (I was literally unable to sleep and felt zoned out the whole day).

The situation did not improve and I was on my way to the emergency room (I had no other choice); fortunately by speaking with my coworkers found a dermatologist willing to see me in person.

That was NOT the end of it. I am not going to go too much into detail, but here’s the history of the visits I had (so far):

Date	Physician	Diagnosis	Prescription
Apr 1	TelaDoc	Bed Bugs	Triamcinolone
Apr 13	Walk In Clinic	Seasonal Allergy	Claritin
Apr 16	TelaDoc	Hives/Urticaria	Prednisone
Apr 20	TelaDoc	Hives/Urticaria	Hydroxyzine
Apr 23	Dermatologist	Folliculitis/Atopic Dermatitis	Keflex/Clobetasol
Apr 30	Dermatologist	Folliculitis/Atopic Dermatitis	Keflex/Clobetasol
May 15	Dermatologist	Skin infection	Not disclosed until I know more
Jun 3	Dermatologist	Skin infection	Not disclosed until I know more
Jun 12	Dermatologist	Skin infection	Not disclosed until I know more
Sep 11	Dermatologist	Skin infection	Not disclosed until I know more

You can see that almost every doctor had a different idea of what my problem is. It seems like the last one finally got it right. I’m still keeping my fingers crossed.

During this weird situation I discovered:

I am effectively allergic to Cephalexin
My family has an history of Psoriasis
My sister has been suffering of atopic dermatitis for a while

What about now?

My skin issue is not solved yet, although it seems that it has been dramatically improving.

In any case, I am now trying to leave the country on July to rest a bit in Europe, and try to start over in September. Again.

Frequently Asked Questions

Vincenzo Chianese — Fri, 29 May 2020 11:15:57 +0000

This series is about sharing some of the challenges and lessons I learned during the development of Prism and how some functional concepts taken from Haskell lead to a better product.

Note: As of January 2021, I no longer work at Stoplight and I have no control over the current status of the code. There is a fork on my GitHub account that represents the state of the project when I left the company.

While explaining the journey of refactoring Prism around and writing this article, I've received a set of FAQ. I've grouped them here.

What’s the state of the functional rewrite?

I am generally satisfied with the shape of Prism; but this journey is not over yet.

At first, not all the parts have been converted/refactored to use fp-ts. Although this series is only talking about that, I want to emphasize it has never been our primary focus. We never halted the regular development of Prism to rewrite its parts; on the contrary we have continued to fix bugs and ship new features. We never broke the user space.

There are still good opportunities to refactor and make the codebase even better. To give you a brief idea of what we’re working on at the moment:

We recently introduced the Do notation (lend from Haskell) to make the code even more readable, with everybody's quick approval on that: https://github.com/stoplightio/prism/pull/1143
We will hopefully start to work on the third validation refactor, where I hope to introduce a new data structure (These) that will allow us to have an accumulating (and not halting) validation in case of warnings

On the other hand, there are some parts that will probably never be refactored to be functional, because the value that it would bring is less than the cost of making the transformation. About this, one good example is logging: as I mentioned in part 3, logging is deterministic but it has side effects; therefore, it should be wrapped in a IO monad.

I do not see this happening. This is one of the tradeoffs and I think it is important to stay pragmatic; I am not one of these programmers that are feticist about functional concepts.

Sometimes I had to intervene to stop the FP discussion from going too far: https://github.com/stoplightio/prism/pull/649#discussion_r329107225

What do your coworkers think about it

Believe it or not, so far all the people that have been working on Prism — after some time — ended up loving it. What I have observing with all the people that had to work on Prism (and some other software, since I’ve been in meantime expanding the usage of this into the internal Stoplight’s codebase — is that there are essentially 4 phases the people go into:

What the heck is this
I understand it, but I do not like it at all
Oh now I get why this is useful
I think I am in love and I want to write all the software with it

I went into the same exact steps listed here but more importantly I do remember going through the same phases also when I had to use React the first time — and recently when I started to use TypeScript professionally.

Hopefully this is a good evidence that functional programming and its related tooling has no difference with any new methodology/piece of technology: people are just scared about new stuff and with the good amount of education they will go over it.
At the time of writing, I have people in phase 4 as well as in phase 1 and 2.

There is a difference with adopting React or TypeScript though. The first two have a very wide audience and online you can consult. It is easy to find people familiar with the technologies.

Although functional programming has been around for way more years than React or TypeScript, we have to face the fact that’s not that spread around as some of us want.

If we pair this with JavaScript, then the niche is already significantly narrowing and if we add TypeScript on top of it, we start to run out of options. For this reason, I was essentially the education for my comrades. Here are some examples of conversations we had:

https://github.com/stoplightio/prism/pull/648#discussion_r328157183
https://github.com/stoplightio/prism/pull/648#discussion_r328160675
https://github.com/stoplightio/prism/pull/648#discussion_r328162406
https://github.com/stoplightio/prism/pull/648#discussion_r328165087

You can see that these comments, more than code reviews, were more like live tutorials on the code. From my point of view, they have helped my fellow coworker speed up the onboarding significantly. It also made him excited about it

How much time did it take to get this far? Was it worth it at all?

Giving a precise timeline is hard, since we never stopped completely working on Prism to refactor the codebase. This has always been a parallel and opportunistic work. Just looking at the dates though, we started in June last year and we still haven’t finished it yet.

On the worthiness of the whole operation your mileage will of course vary. I still have people in the company that did not even dare read the codebase and just claimed it’s bad, but I firmly believe the quality maintainability of the code outweighs pushing contributors away and I’ve stayed away from these kinds of conversations. There are some points that we gained with the switch that alone, were worth the efforts.

I have never seen Prism crashing on my computer. I’ve never seen Prism crashing on our servers in the hosted version. I’ve never seen a bug report about a crash. Sure, it will respond incorrectly from time to time — but that is a completely separate issue. Since all our errors are modeled as Either, there is no way you can forget to handle an error making the software crash
We as a team are always automatically on the same page. There’s no more debate about throwing an exception vs return null vs return undefined and then try to handle all the use cases somehow. There are a lot of areas where the appliance of functional concepts just makes everybody agree. We have only one rule: if it composes, then 99% it’s good. If it does not, then something is wrong.

During this journey and while telling the people about it, I received some questions multiple times. I’m going to try to answer all of them here.

Why not using Haskell/F#/Clojure/WhatEverFancyLanguage?

This is a non-question for me. I am familiar with Haskell and for sure, I would love to have Prism in Haskell. It would probably be an even better product.

On the other hand we have to stay pragmatic and Stoplight made the initial investment in TypeScript — and such language is here to stay. This does NOT mean that I can’t write good software though.

In particular, I got the feeling that TypeScript is mature enough to give a non optimal, but still a pretty decent experience when writing functional code. With Prism, I finally have a tangible example when people are pointing out that this is not possible and we’re condemned to write shitty code forever and ever:

Do you think FP makes always sense?

That’s also kind of a non-question too. I have seen what I call feticist that are like “everything is functional or you’re out of the game” — or something along these lines. I do think it is possible to stay pragmatic and grab the abstractions you need for your application.

For instance, in Prism the functions emitting logs are considered pure, even though they clearly are not (if you remember from the previous article, console.log is deterministic but it has the side effect to write on the screen). This is theoretically wrong, but for the sake of my application, I really do not care.

I will say though that it is always going to make sense to model a significant class of errors not as an exception, but as a real entity of your domain model. Making your application error-aware is only going to give you benefits.

For instance, when looking for an user via email in a database — the fact that such a user does not exist is very possible. There is no reason to throw an exception for that instead of returning an Error object that people will have to handle accordingly. The choice of the errors to treat in such a way is ultimately up to you.

In the case of Prism, we are kind of lucky since it has almost no interactions with the outside impure world (file system, network) and when they happen, most of them are confined in the CLI. The core of Prism is pretty much pure and functional and so almost all errors are properly modelled. Functional core, imperative shell. In a line of business applications though, things might be different.

Are you seriously betting all this on an unknown library called `fp-ts`?

The short answer is yes. In general I have never been hype or GitHub star driven, and for this case I do not care about the library itself at all.

The reason is simple: fp-ts and any other alternative that you can find on the web (another notable one is funfix) are simply formalising a mathematical structure. It’s a set of proven laws that, unless some mathematician wakes up one day and claims “We did it all wrong for the last 200 years” — ain’t going to change.

What about the performances?

There’s always going to be somebody in the audience asking this question, and my feeling is that somebody asking this is probably missing the point of the whole presentation.

In any case, since I was getting this so much I decided to collect some random data and see what the results would look like.

I am not going to go too much into details, but essentially by looking at the flamegraph of Prism responding to 10k, it turns out the bottleneck is mostly in the validation and the example generation. I was barely able to find any overhead driven by the monads used in Prism. I have never ran any memory benchmarking, and I am not planning to run one at the moment.

Thanks!

If you've arrived here, it means that you've probably enjoyed the whole series and I hope it brought you some value.

I wanted also to thank everybody that's been proofreading all the parts and did some constructive observations and comments. I would have to put so many names that it's probably better to just say thanks to all.

The road to the return of the investment

Vincenzo Chianese — Wed, 27 May 2020 10:29:29 +0000

This series is about sharing some of the challenges and lessons I learned during the development of Prism and how some functional concepts taken from Haskell lead to a better product.

In the previous post I introduced fp-ts in Prism with the logging as the primary use case. In this post we'll take a look at how the usage of fp-ts slowly spread in the whole codebase, how we misunderstood some concepts, how some of the coworkers took the adoption of fp-ts and how it helped us refactor problematic parts.

First Expansion: Router

Time passed after the merging of the PR introducing fp-ts in Prism; in the meantime teams in Stoplight were reshuffled a little bit. As a result, I got a new teammate on the project. Curiously, he was previously working on the initial new Prism design; then he was reallocated somewhere else when I took Prism and now he was coming back.

Essentially, I had a fresh member to onboard on the new direction I wanted to give to the code base. I quickly realized this was an incredible occasion to show the company that picking up functional concepts is not an impossible mission and I wanted to play my cards in the most efficient way.

As the first step for the onboarding, I decided to let my new comrade review a PR I would write that would migrate a component to a functional approach.

From there, I would then observe his reactions and of course answer his questions.

This time identifying the next possible candidate component to refactor was easy. As I explored in the part 1, Prism has the following components:

Router
Input Validator
Negotiator
Output Assembler
Output Validator

The negotiator was partially done already in the first PR introducing fp-ts in Prism, and I was well aware that the validation (both input and output) would require a major refactor since they were all state-class based and objectively complicated more on this later.

I decided to go with the router. Being the first part in the whole flow it would have almost no dependencies from the previous steps, meaning that there would not be plumbing code and/or weird wrappers to match inputs and outputs. Further, its logic was not complicated and the refactor was exclusively to bring it into the functional world, with no changes on its behaviours; this way my comrade would only review effective fp-ts related changes.

Expand fs-ts in Prism's router #402

XVincentX posted on Jul 06, 2019

The following PR extends the usage of fp-ts to the routing package as well by basically making sure it does not throw exceptions anymore, but rather use the Either object to express an error object.

With this — the router and the mocker finally compose because the type match (they both return an Either<Error, T>.

Extend the Either usage to the router was indeed the easy part: https://github.com/stoplightio/prism/pull/402/files#diff-f9a10b37616fb5669ecd5218fc8535c9L16

The problem started when I started to integrate and try to compose the new function in the mega-file-to-split:

The whole flow is sync apart from the edge case when we need to employ the forwarder, and this requires an additional abstraction layer https://github.com/stoplightio/prism/pull/402/files#diff-47c5dc2d65fd624c869f5f08d0cfb56aR45
What's really preventing from having a clean and functional flow is the validation process that's basically creating an empty array, giving it to the mocker and expecting to receive a filled array. This forces me to keep some stuff here and some stuff there; if the mocker could just return the validations, that'd improve the code a lot.
In order to keep the API compatible with what we have, I have to do some wrapping I'd like to avoid https://github.com/stoplightio/prism/pull/402/files#diff-47c5dc2d65fd624c869f5f08d0cfb56aR98

That said, the funny thing is that, although this Pull Request is meant to be an improvement, you can argue that the code is effectively uglier than it is. (Well I do not think it is, but you mileage may vary)

The good news though is that — I'm not sure if you remember, we were discussing on how to refactor this part and nobody (me included) really came up with good ideas.

By trying to extend the functional parts to the router — I now know exactly what needs to be done and how to move forward. This is freaking awesome, to be honest.

View on GitHub

The conversation was not that long and chatty as the first one. I also remember there was almost no conversation at all in our internal Slack channel.

It is hard to tell why exactly. It could either be because the team assimilated the concepts or maybe they "resigned" on the fact that this was happening and so arguing would not have changed a lot.

I find the first one very improbable and I would say the truth is in between but clearly leaning on the latter hypothesis. The regret I have today is not asking this explicitly instead of taking advantage of the situation to merge the PR in right away.

My teammate observed:

the code looks complicated because it is long and deeply nested

It is interesting because the code has been long and deeply nested since forever. fp-ts made that thing visible to a point it could not be ignored anymore. We’ll see an example of a successful refactor later.

The feedback I was receiving in our internal channels was that generally the code would look dirtier than it was previously. This was mostly happening at the "edges" between the regular code and the functional one because of required bridging to maintain compatibility with the current Prism functionality.

For instance, there was a function of Prism that could have thrown an exception in case something went wrong.


function foo() {
  // a lot of code
  if (!condition)
    throw new Error('This is not ok');

  // a lot of code again
  if (!anotherCondition)
    throw new Error('This is not ok');

}

When such a part got refactored to use Either<Error, T> exceptions wouldn't be thrown at all. On the other hand, whoever would have called this function might have relied on the thrown exception. As long as all the callers wouldn't have been refactored, foo would always have to ultimately throw. This is what I called "bridging".

For the foo function, the bridging would probably look like this

import * as E from 'fp-ts/lib/Either';
import { pipe } from 'fp-ts/lib/pipeable';
Import { identity } from ‘lodash’;

function foo() {
  pipe(
    operation1(arg1, arg2),
    E.chain(result => operation2(result.outstandingBalance)),
    E.chain(operation3),
+   E.fold(error => { throw error }, identity)
  );
}

There were cases with some monads where the bridging code would look even uglier. On the positive side, this would clearly communicate to the developer that this function was still impure exclusively because of something relying on the impure behaviour. This facilitated the search for refactoring opportunities significantly.

Return Of Investment: Validation

Finally after some time we got the downpayment of a series of returns of investment given by the employment of fp-ts in Prism.

I have already stated that validation in Prism is hard, and the way it was initially implemented in Prism made it even harder. We complained and tried to do something about it (with no results) multiple times:

https://github.com/stoplightio/prism/pull/351#discussion_r293254707

You can see that ultimately the whole team would agree that passing on the opportunity would be the best idea for the time being, since it would be too much time consuming.

The real deal was that nobody knew where to start. That piece of code was terrible, but fp-ts gave me the key to move on and finally refactor that part of the validation.

One of the good things when using category theory constructs is that things tend to naturally compose. It's like having two pieces of code with a magnet at the extremities: they naturally want to bond. Such property suggests to you that, when things do not compose, something is probably not going well.

Let’s take yet another look to some of the Prism components:

Router (fp-ts-ized)
Input Validator
Negotiator (fp-ts-ized)

We were fundamentally in the situation where two pieces that wanted to compose (the router and the negotiator) couldn't because the Validator had not the right interface. The lack of composability became the driving factor that I used to refactor the input validation.

What happened was fascinating: I was about to ask for suggestions in the Slack channel where I used (and still use) to hangout and talk about functional stuff. While writing the message, I wrote the solution without getting any input from outside:

The last sentence I wrote is kind of memorable

I think fp-ts made me understand why the code is so horrible and how I should move forward.

This, indeed, ultimately happened some time ago:

https://github.com/stoplightio/prism/pull/862

When It Went Wrong: Security Checks

This is not a story where we did everything right. It would mean that's either invented or it omits details. Although I am inclined to say we did most of the things right, we clearly made some mistakes along the journey.

One of these was the porting of the security checks. It's essentially the part that checks if the call can go through the negotiator for mocking or being rejected with a 401.

This part didn't go very well. Although after the conversion the feature was still working correctly from a functionality standpoint, the resulting code was really hard to reason about, resulting in two additional refactors to bring it back to a reasonable state.

Looking at this now I think there were some factors that brought the things out of control:

Although familiar with functional concepts, the developer working on the security checks didn't grasp enough of it to be able to complete the feature alone. This resulted in frustration on his side and ultimately brought him to switch in the “get it done, no matter what” mode.
I also had some blanks. For instance, I thought that passing a Left<T> as a function argument would be totally legit — it turns out that 99% it is not.
In order to get the feature done and give some relief to the comrade, I lowered my code review standards and merged it in anyway.

This episode costed me a lot of time to clean it up:

https://github.com/stoplightio/prism/pull/804
https://github.com/stoplightio/prism/pull/830

The second PR, although the changes are minimal, took me ages to put together. I still have a vivid remembering of me working on it. The logic behind that was so complicated that I would lose the context quickly and had to restart from scratch. Multiple times.

What are the lessons learned?

It is inevitable that things will go wrong during any journey. Keep that in consideration and allocate some time to clean up stuff.
Short term solutions will bring long term conflicts. Short term conflicts will bring long term solutions. I decided to give my coworker a relief by merging something that wasn’t really ok. I had to pay that back with a very high interest rate.

In the next article, we'll respond to some of the FAQ that I have received while talking about this and showing Prism around.

Introducing in the company

Vincenzo Chianese — Fri, 22 May 2020 12:48:25 +0000

This series is about sharing some of the challenges and lessons I learned during the development of Prism and how some functional concepts taken from Haskell lead to a better product.

In the previous post we saw how I discovered a set of abstractions that apparently were a perfect fit for Prism's use cases. In this post we'll instead cover how we got the concepts applied for the first time in Prism.

Finding an introduction point

At this point, I was convinced that adopting fp-ts in Prism would make a lot of sense; the problem was to find a way to introduce it in a reasonable way.

Prism was already at a late development stage. Although we didn't have a fixed timeline we were already releasing beta versions a couple of times per week and we also had a decent amount of users trying it out.

For these reasons, I decided there were a couple of points that I was not willing to compromise while trying out fp-ts.

Although we did not have a precise deadline, I knew that Prism's stable release was close. For this reason, no matter what was the state of fp-ts in Prism, I set myself the goal that the master branch should always be ready to be released as the new major version.
Because we already had a good number of users, I set the hard rule that we would never break the user space
Because I knew that this was new stuff even for me and nobody in the team was really into functional programming, I set the aim of introducing this into a relatively low-risk part of the software so that:
- It would be relatively easy to revert the changes in case we'd realise that fp-ts wouldn't really fit the use case or in any case not play well together with the rest of the code base
- In case we would have misunderstood some parts of the library or screwed up something, it should not alter the software in any way. Preferably, the main features of Prism should still be functioning correctly

These are very strong constraints, narrowing down the possible choices a lot. I guess that's the fun of the software modelling.

Eventually the occasion I was waiting for came up.

Logging in Prism

Logging is probably one of the most appreciated features of Prism because it provides a clear introspection on the decision process and tells you exactly why Prism has decided to respond to you in such a way.

Moreover, the logger brought a lot of value to us as well. If for some reason Prism would respond incorrectly, because of the logger being so verbose we've always been able to quickly identify what component was misbehaving (negotiator, validator, the http server)

Thinking about the logging in Prism I realised that:

There would be no way the logging would break the user space, since there was no impact nor in the CLI, nor the Prism HTTP Client API nor the Http Server
The worst that would happen in case the logging would be wrong (because of a misusage/misunderstanding of fp-ts) would have been wrong log lines; the mocking functionality wouldn't have been affected per-se
At the time of the introduction logging was planned only for the Negotiator, making possible an ideal "revert PR" and return back to where we were.

For these reasons I thought the logging was a good introduction point and I took the risk of trying it out fp-ts.

Getting the PR merged

I decided to go with pino as a logging solution. The decision was NOT based on the low overhead/high performance claims that are on the website/project documentation, but more because Pino does not include any way to process/alert/react to logs; it simply outputs the log statements as JSON on the stdout (or any stream you provide to it). It means that, by design, there's no way somebody can make messes with logs and do any action that's logging dependent. You can't break what's not even included.

Now the compromises start to come. In the complete pureness of the functional world, writing messages on the standard output through console.log is deterministic (since it always returns undefined) but it has a side effect (printing on the screen).

Haskell indeed considers it as an action that can only be ran in the main program

putStrLn :: String -> IO ()

Similarly, fp-ts provides the same mechanism:

export function log(s: unknown): IO<void> {
  return () => console.log(s)
}

Moreover, printing on the screen is not considered something that it might fail – so the Either abstraction (which is the one I really valued in Prism and wanted to adopt) would not be useful in this specific case.

We'll talk more about the compromises in a next article; however I decided, at least to start that:

I would not consider the console.log as a side effect; therefore the actions would have been considered pure at the time. I was perfectly aware that it's wrong, but to respect the rules that I gave myself (and listed above) I had to go with a minimal codebase invasion
Since logging is not considered to be a failing action and I would not consider it to have a side effect, I decided instead to get started with the Reader monad.

The idea was to use the Reader monad to inject the logger to the Prism Http Package without having to pass it explicitly as a parameter everywhere.

This made a lot of sense, since we wanted to have different loggers according to the context (CLI, Hosted Prism).

I had a very brief conversation about my intentions with one of my coworkers (who essentially nodded) and I made it happen:

Logging Prism's decisions. #323

XVincentX posted on May 27, 2019

All right, be prepared.

TL; DR

Prism can now log stuff from the negotiation.
The negotiation and logging process can't make Prism crash anymore for any reason.

Long version

The following PR, in theory, implements logging for the negotiator. Practically, though, this PR is laying down the foundation for the logging for the future hosted version as well and hopefully start moving Prism's code in a different direction.

There were some main principles I've been keeping in mind while writing the code for this stuff:

Prism should not process/format/react to the logs. Somebody else should do it; doing stuff on logs is usually slow and Prism should instead answer all the requests as fast as possible.
The negotiation process in Prism is currently composed by numerous and nested function calls and last thing I wanted to do is to carry over this logger parameter in all the function and having to deal with it everywhere
The logging cannot be a singleton defined somewhere — because the logging instance will be provided externally (Prism Http Server, in this case)
The logging process and the negotiation process should never make Prism crash. In any circumstance. I repeat, in ANY circumstances.

Let's now see how I've tamed them down.

Use Pino. If you go on their website the write that "it's the fastest logger on the world" but you can ignore that, that was not the reason why I went with it. The main points were:
- It's included with fastify — and so we're not introducing a new dependency. We've been shipping it since forever.
- Pino does not include any way to process/alert/react to logs; it simply outputs the log statements as JSON on the stdout (or any stream you provide to it). It means that all the processing must be off process and I think this is great. It means that, by design, there's no way somebody can make messes with logs and do any action that's logging dependant. You can't break what's not even included.

So where's the log processing happening in our case? In the CLI. You can see from the code that, whenever the production environment is detected (or the -m flag is passed) the CLI will effectively use the cluster module to fork the process, run Prism Server and funnel its stdout logs back to the CLI, where Signale will pretty print them. When Prism will be hosted somewhere, we'll likely do the same.

In case you're running the things locally to write code and test the stuff, you'll see that the fork is avoided and all is done in process. This is done fundamentally because it's going to be way easier to debug stuff in a single process. This is exactly the same that Jest does with the --runInBand flag.

You can see from the image what when the -m flag is specified, you'll find an additional process with node running — this is exactly the forking happening.

Note: if NODE_ENV===production the forking is enabled by default, so that if you use the CLI in a Docker Environment you get what the production behaviour.

Inject the logger through currying. Thanks to the partial application I can basically write a function that takes the regular arguments which returns a function needing an additional dependency to run. const negotiate = (arg1, arg2, arg3) => logger => { // real body }. For this, I've employed fp-ts's reader implementation, that allows me to do exactly that: write code that has regular arguments and returns something that will need a logger to be executed. By leveraging chain and map method, intermediate functions can easily introspect and modify the result in the middle without having to worry about the logging presence. It'll be put as last step. Moreover, if one day we will want to make the Router start outputting logs as well — it's really easy. All we need to do is make the router spit up a Reader and chain it with the following one in the mocker. All good, profit.
See point 2. With Currying I can pretend the inner functions will eventually have a logger, it's resolution is happening in a completely different npm package; this is important to understand why I could not simply have a shared instance to require from somewhere.
This happened as an effect of the reader. Since the logger in injected at the end, the traditional try catch in the middle of the code to decorate the errors does not work anymore. I could have workaround this but I decided instead to change the approach and make sure the negotiation and the logging process never throws exceptions. Therefore the code has been modified to use the Either implementation that's included in fp-ts, which is not that hard to understand. If you check out the NegotiatorHelper.ts you can see there are not that many changes and more importantly nested exceptions are now been flattened — which is good. Clearly there are 1-2 things that aren't that cool but I'm on all of them. In particular:
- Jest should support wrapped helpers so I can remove my home made helpers https://github.com/facebook/jest/issues/8540
- A built-in ReaderEither could let me remove some nested maps. I'm quite sure @gcanti will make it happen soon

Outstanding points

[x] Refactor mock.ts to look a little bit more human
[x] Resolve the null-ness of logger component
[x] Understand whether we're logging enough or too much
[x] CLI Readme Update
[x] Deduplicate the testing helpers
[x] General review and cleanup

SO-231

View on GitHub

The PR is kind of big and although I would encourage you to check it out, I am pretty sure nobody will. Therefore, here's the TL;DR:

I decided to put my --verbose flag to 9000 and over-explain my intentions, how I executed them, what were the consequences, limitations and how we would move forward with that.
Some coworkers were confused by the terminology; people unfamiliar with the functor in general will think that map is only a thing for arrays and make up their noses when they see it applied somewhere else
A coworker was excited about the work and since he effectively was a little bit more familiar with this stuff, he helped the review and corroborated the benefits that I was trying to bring
Different libraries and languages have different names for the same "operation". In the specific case, we had people confused by the chain function because somewhere else is usually called flatMap or bind
We had some people concerned about the general readability of the code (that's a funny thing to me but I was aware that people's brain needed to be rewired) and growing the contribution barrier

I took all the necessary time to address the comments; most of them were actually more questions and clarification rather than requests for changes. Most of them were easily resolvable once I would briefly explain the concept behind that (such as chain, map, fold). I took that as a good sign.

Then the conversation kind of stalled for some days until I decided to take control of the situation and risk it by merging the PR although not everybody approved it. I felt that in some cases it is better to do something and ultimately be wrong rather than not doing anything and then be wrong anyway. Nobody complained about it so it was official, we were starting to get the core of Prism functional.

Somebody was a little bit angry about it, but I also received a great feedback during one of our retrospectives:

The introduction of fp-ts in Prism makes me think we'll finally do something differently here.

In the next article we’ll see how, by continuing to expand fp-ts’s usage in Prism we started to get the first wins and return of the investment we did.

Finding the right abstraction

Vincenzo Chianese — Fri, 15 May 2020 12:59:22 +0000

This series is about sharing some of the challenges and lessons I learned during the development of Prism and how some functional concepts lead to a better product.

In the previous post we had a 30k feet view of Prism and some of its technical features and challenges.

In this specific post, I will explain the series of events that ended up me deciding to give a try to a functional library that apparently would fit the Prism's use case perfectly.

A validator in TypeScript

Prism was not the first project that I was working on when I joined Stoplight; before landing there I have been bounced around different projects while trying to understand where I could best contribute.

During that time, the development of the new version of Prism was just getting started by a different team, and I was totally unaware of the project's status, direction and technical choices that were made.

However, we can say that Prism was developed as a typical TypeScript web application; throwing exception where things were going unexpectedly wrong (somewhere returning null or undefined), using classes holding state for the validators, a strong classical inheritance model to let the software expand in the future and so on.

Months pass. I am working on Stoplight Studio and I was actively looking for a validation library that would be TypeScript friendly and would be able to type the returned value in case the validation would pass.

I googled the relative keyword around ending up in solutions using decorators that were not what I was looking for.

I do not like decorators. They're not even standards. I do not like classes neither.

During the search though, I stumbled on this Tweet that gave me some hope:

// Detect dark theme var iframe = document.getElementById('tweet-1123930225857974272-105'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1123930225857974272&theme=dark" }

I headed to the mentioned project and after a quick read to the README (which, I would agree with you, is not that…eloquent) I felt it looked promising and so I tried create a sample project to try it out.

First impressions

Note: this image is relative to io-ts 1.x. The current version is 2.x. Your experience is likely going to be different.

There are two things that are very evident:

There's a bunch of methods whose name aren't familiar at all:
- inspect
- isLeft
- isRight
- map (well actually this was not unknown)
- mapLeft
- orElse
- reduce
- …the list goes on and on
I can't find my data. Once I've defined a type and decoded an unknown payload, where is my data stored? I could only see functions.

I was already on the point of ditching the whole thing and look for the next alternative, but I then noticed I actually knew the author (we've been hanging out in the same Slack channel for a while, but mostly ignoring each other), so I decided to ask what was the deal with io-ts.

After some chit chats, it turned out that io-ts is using fp-ts, providing the functional primitives that the library is using and exposing, and that was the primary reason for the "weird" set of functions returned from the decode process.

Ultimately we ended up philosophically discussing what kind of software I have been writing all these years by ignoring the functional concepts.

At the end of the day, I learned that the decode function of io-ts would return an Either — a monad representing the effect that a computation might fail. So, in order to continue my journey, I had to master that.

It's not my intention here to explain to you functional programming. I am not qualified for that and if you're interested I really suggest you to read the amazing series of articles written by Giulio Canti directly:

Giulio CantiFollow

/gcanti

However, I just wanted to give a basic explanation of the Either monad and how I felt it would be just what I'd need for Prism

Either 101

An Either is a tagged union that holds a value whose type depends on the tag property.

type Either<E, A> = Left<E> | Right<A>

interface Left<E> {
  readonly _tag: 'Left'
  readonly left: E
}

interface Right<A> {
  readonly _tag: 'Right'
  readonly right: A
}

What you want to store in such a structure it's ultimately up to you, but what's mostly used for in all the code I saw so far in the wild is essentially to hold errors (usually put on the Left) and the value (usually on the Right) in as an alternative way to throw an exception when something is going wrong.

For instance, consider this function:

function createParamUriTemplate(name: string, style: HttpParamStyles, explode: boolean) {
  const starOrVoid = explode ? '*' : '';
  switch (style) {
    case HttpParamStyles.Simple:
      return `{${name}${starOrVoid}}`;

    case HttpParamStyles.Label:
      return `{.${name}${starOrVoid}}`;

    case HttpParamStyles.Matrix:
      return `{;${name}${starOrVoid}}`;

    default:
      throw new Error(`Unsupported parameter style: ${style}`);
  }
}

and now let's rewrite it to use the Either tagged union:

import { left, right } from ‘fp-ts/lib/Either’;

function createParamUriTemplate(name: string, style: HttpParamStyles, explode: boolean) {
   switch (style) {
     case HttpParamStyles.Simple:
+      return right(`{${name}${starOrVoid}}`);
-      return `{${name}${starOrVoid}}`;

     case HttpParamStyles.Label:
+      return right(`{.${name}${starOrVoid}}`);
-      return `{.${name}${starOrVoid}}`;

     case HttpParamStyles.Matrix:
+      return right(`{;${name}${starOrVoid}}`);
-      return `{;${name}${starOrVoid}}`;

     default:
+      return left(new Error(`Unsupported parameter style: ${style}`));
-      throw new Error(`Unsupported parameter style: ${style}`);
   }
 }

Such changes inevitably will mandate a change in the function signature:

import * as E from 'fp-ts/lib/Either';

 declare function createParamUriTemplate(
   name: string,
   style: HttpParamStyles,
   explode: boolean
- ): string;
+ ): E.Either<Error, string>;

And it also made me observe something:

While the first signature does not communicate in any way that the function could error and throw an exception (the only way to get that is to inspect the source code), the second version is very clear about the possibility of a failure.
While in the first case I can pretend (and pray) that no error will happen and carry on the happy path only, the second function will not let you access the data if you don't perform explicit checks
It makes the Error an explicit part of your domain model instead of an exceptional thing that should never happen (that WILL happen one day, believe it or not)
Since Either admits instances for Monad, Applicative and a Functor, it can leverage some good tooling to restore composability even if processes that might error (more about that later)

Obviously, these points didn't come in my mind all at once; it required some effort and education to get past the "meh, does not look so good" phase and understand the concepts and the benefits it would bring (and make the coworkers excited about it, but we'll discuss this later).

I was still super skeptical about it, but the bell was ringing. At that time, I had the two considerations in mind:

If Prism's job is to return errors then treating them as a domain entities instead of a undesired effect that should never happen (aka: exceptions) would make a lot of sense
By having to handle all the errors no matter what, I could make sure that I would always return the correct status code and not leave anything unhandled (and returning an incorrect 500).

It was worth giving it a try; but how do you introduce such stuff in a codebase that's already in progress, with other people working on it and that's close to a major release?

We'll explore that in the next article.

Forewords and domain model

Vincenzo Chianese — Fri, 08 May 2020 11:16:27 +0000

This series is about sharing some of the challenges and lessons I learned during the development of Prism and how some functional concepts lead to a better product.

In this specific post, I will start explaining what Prism is, detail some of its key features and discuss a little bit about its domain and its intended audience.

This will hopefully help you understand the technical choices I made that I will cover in the next articles.

What Is Prism

stoplightio / prism

Turn any OpenAPI2/3 and Postman Collection file into an API server with mocking, transformations and validations.

Prism is a mock server for OpenAPI 2 (from now on OAS2), OpenAPI 3 (from now on OAS3) and Postman Collections (from now on PC).

For those of you that aren't familiar with such, OAS2/3 and PC are essentially specifications defining a standard and language-agnostic interface to (possibly RESTful) APIs.

To be a little bit more pragmatic:

openapi: 3.0.0
paths:
  /list:
    get:
      description: "Returns a list of stuff"
      responses:
        '200':
          description: "Successful response"

This YAML file is an OpenAPI 3.0 document claiming that:

There's an API
It has a /list path
It has a GET method
When a GET request to the /list endpoint is made, 200 is one of the possible responses you might get, whose details (such as payload shape, returned headers) haven't been specified.

We aren't going to go too much into detail about these formats; if you’re interested, you can go and read the official specifications:

Despite this simple example, we can say that all the specifications allow (with some nuances) to specify pretty complicated scenarios, ranging from authentication, request and response validation, to web hooks, callbacks and example generation.

A mock server is nothing more than a little program that reads the description document and spins up a server that will behave in the way that the document mandates.

Here's an example of Prism starting up with a standard OAS3 document:

Prism Peculiarities

Technical decisions and trade-offs were driven by features. Here are the most relevant ones regarding this series:

100% TypeScript

Prism is written entirely in TypeScript. Primarily because Stoplight's stack is largely based on NodeJS and TypeScript.

We are using the maximum level of strictness that TypeScript allows.

A Lot Of Custom Software

Prism is not using any of the web frameworks you usually find on the market and employed for web applications, so you won't find Express, you won't find Hapi, nothing.

It was initially written using Fastify; and at that time I was not working on the project. I ultimately decided to remove it in favour of a tiny wrapper on top of the regular http server that NodeJS offers.

In case you are asking, the main reason for this is because most of the frameworks focus on the 80% of the use cases, which is totally legit.

On the other hand, Prism aims for 100% compatibility with the document types it supports, and, for instance, some of them have some very…creative parameters support that no parser on the market supports.

Another example? OpenAPI 2 and 3 are using path templating, but not the same as URI Templating specified in the RFC6570. For this reason, a custom parser and extractor had to be defined.

This specific case, together with other ones that required special code to be wrote, led us to gradually dismantle and neglect different Fastify features until I realised that we were not using it at all if not for listening on the TCP port; on the contrary, we were just fighting it because it was too opinionated on certain matters, such as errors.

You can find more about the motivations in the relative GitHub issue

Custom Negotiator

Prism contains a custom made negotiator — which is that part of the software that taken an incoming HTTP Request, its validation results (headers, body, security) and the target API Specification document will return the most appropriate response definition that can then be used by the generator to return a response instance to the client.

The negotiator itself is kind of complicated, but I think we've done a good job in both documenting its decision process:

The diagram is also pretty much reflected in the code as functions division.

Input, output and security validation

One of Prism's key features is the extensive validation.

Based on the provided API Description document, Prism will validate different parts of the incoming HTTP request, ranging from deserialising the body according to the content-type header and then checking the resulting object with the provided JSON Schema (if any).

The same goes for the query parameters (because yes, OpenAPI defines encoding for query parameters as well), the headers and ultimately the security requirements.

The input validation result will influence the behaviour of the negotiator as well as the proxy's one.

It turns out that validation is a very complicated part of Prism and, although we have reworked it several times we still haven't got that right.

Prism Request Flow

The journey of an HTTP Request from hitting your application server to return a response to the client is articulated.

We often do not think about it because the web frameworks do usually a very good job in abstracting away all the complexity.

Since Prism is not using any frameworks, I fundamentally had the opportunity of reimplementing almost the whole pipeline — and I started to have observations.

Here's what Prism is doing when a request is coming in:

Routing
- Path Match with templating support, where we also extract the variables from the path, returning 404 in case it fails
- Method Match, returning 405 in case it fails
- Server Validation, which is checking the HOST header of the request against the servers listed in the specification document, returning 404 in case it fails
Input deserialisation/validation
- The path parameters get validated according to what is stated in the specification files (whether it's required, whether it's a number or a string) 422/400/default
- The query string is deserialised following the rules stated in the specification file, returning 422/400/default in case there is a deserialisation failure
- Headers get validated against the JSON-esque format that OAS2/3 defines; we convert them to a draft7 specification and run ajv on it, returning 422/400/default in case there is a validation failure.
- Body gets validated against the JSON-esque format that OAS2/3 defines; we convert it to a draft7 specification and run ajv on it, returning 422/400/default in case there is a validation failure.
- Depending on the security requirements specified in the routed operation, Prism will check the presence of certain headers and when possible it will also try to validate that their content respects the general format required for such security requirements. Returns 401/400/default
Negotiator/Proxy
- The negotiator kicks in and looks for an appropriate response definition based on the validation result, the requested content type, the accepted media types and so on. It returns 2XX/406/500/User Defined Status code depending on the found response definition.
- If the Proxy is on, Prism will skip the negotiator and send the result to the upstream server and take note of the returned response.
Output violation and serialisation
- Response Headers, whether they're generated from a response definition, extracted from an example or returned from a Proxy request get validated agains the response definition, returning 500 (erroring the request or a violation header) in case they do not match
- Response Body, whether it's generated from a response definition, extracted from an example or returned from a Proxy request, gets validated agains the response definition, returning 500 (erroring the request or a violation header) in case they do not match.

Here comes the first key observation: almost every step that Prism executes might fail, and each failure has a specific semantic meaning and precise status code is associated.

Last time that I checked, on over 32 "exit paths", 30 of these were errors and only two of them were a "successfully returned response". Doing some math:

$2/32 = 1/16 = 0, 06$

This fundamentally says that, in case of evenly distributed exit paths occurrences, only 6% of the request will be successful.

Are the exit path occurrences evenly distributed? Although I do not have a specific answer to that (but hopefully we will, since we're gathering statistics in the hosted version of Prism) — we have some empirical evidence I’ll talk about in the next paragraph that we can keep in mind.

Prism User

Prism is a developer tool and, although it can be used as a runtime component, it is primarily used by API designers and client developers during the development phase of the API.

This is a very important detail since the typical developer that is using Prism has totally different aims from a regular API developer. The following table summarised some the differences that I've identified with an Application Developer

Client Application Developer	API Developer
Clear mission in mind	No idea of what they're doing
Probably read API documentation	Experimental phase
Likely sending valid data	Likely sending garbage
Aims for success	Changes code and spec every second

When you're developing an application, you're likely striving for success — and so you're going to create all the requests you need with likely valid data, likely following the flow indicated in the documentation.

On the other hand, when mocking an API with Prism, you're deep in the design phase. You'll probably tweak the document multiple times per minute (and Prism will hot-reload the document). You'll likely send invalid data all the time because you just forgot what you wrote in the document. You'll try weird combinations of stuff that is never supposed to happen.

We stated some paragraphs before that in case of evenly distributed exit path occurrences, only 6% of the request will be successful.

Now that we have clarified a little bit the typical user of Prism, it's fair to say that the exit paths occurrences are clearly not evenly distributed and, although we can't give a precise number, we can claim that's heavily leaning towards the errors side.

Essentially, when you send a request to Prism, most likely you'll get an error as a response.

After thinking a lot about this, I wrote this sentence that was the key factor to radically change Prism's architecture.

Prism's job is to return errors.

In the next article we'll talk about the abstraction used to model such use cases correctly and how I found it accidentally.

A reviewer's perspective on getting your conference proposal accepted

Vincenzo Chianese — Mon, 23 Sep 2019 07:01:45 +0000

This year I've been organising the Barcelona instalment of APIdays, where I’ve touched every aspect; from CFP evaluation, venue selection, catering as well as speakers' travel and accommodations.

I’ve spoken at 40 events over the last three years. I was rejected - a lot - a whole lot, and in some cases I really did not understand why. Now that I have been on the other side of the event, I have a better understanding of how things work behind the scenes.

Events get a lot of proposals

APIdays Barcelona had 41 speakers from multiple countries and backgrounds. We received 184 CFPs.

If we remove 20% from the total number because of duplicates and people filling the form with invalid data - that still leaves 148 CFPs to consider.

Problem: it’s too time consuming to evaluate every proposal in detail

Being involved in almost everything, it turns out that my original intention of going through all of the submissions one by one was not a viable option; it was very time consuming and I still have a full time job; therefore I had to develop a "fast framework" to decide whether a presentation was worth a serious evaluation.

I ended up checking the following in order:

Presentation Title
Presentation Abstract
A previous video of a speaking engagement

In case a video was not provided, I would then check

Company submitter working for
LinkedIn profile

Failing any of these steps would mean not being evaluated.

Lesson learned: Title and Abstract are the most important filters

No matter what great company you work for or what great previous speaking engagements/videos, GitHub profile or Open Source projects you have on the internet, these were secondary verifications of your persona and get some more context; if the title and or the abstract did not engage the reviewer somehow, the proposal would not make it.

Note: I sometimes took the job position in consideration for one of the criteria; we'll talk about it more in the Work with speakers section.

Problem: titles and abstract might be misleading

Selecting a presentation is hard because the only information you really have is the Title and the Abstract; sometimes you might get a recording because the presentation has been given somewhere else and in that case, BINGO! You have almost everything needed to make a good selection.

Otherwise, you're left with your own judgement.

My original intention was to get in touch with all the candidates to get a better sense of their proposal, but again, this wasn't a viable option.

While it should not have been a surprise, I did make some poor choices where I was mislead by the title and the abstract where it sounded like there was going to be good content, only to realise later that I was accepting a sales pitch (or something not relevant).

On the other hand, I also feel that I missed selecting some very interesting papers because of mediocre descriptions and titles.

Fortunately several presentations that I had accepted, despite looking like they may not be very good, turned out to be great.

Lesson learned (for the reviewers)

You can't always make the right call. You're going to have failures in your selections, as well as great surprises.

Lesson learned (for the speakers)

You might get away with with a "fake" title and a sales pitch (or also something completely different); but rest assured the reviewers will remember you and you won't likely be able to get away with it a second time.

Yes, I've got my own list.

Work with the speakers

There were some presentations where I was like Yes, but… or also No, but maybe….

In these cases, I usually felt it was a good idea to get in touch with the speaker to kick off a "secondary" screening.

The basic idea in this situation was to review who the speaker was working for as well as to check the speakers peers in LinkedIn.

In case the speaker was working for a known company, like Microsoft, Google, GitHub, or any that are "known" in the space, it's possible that the speaker could either point you to the right person in their organisation who had a better expertise/experience in the topic you're looking at or
work with you on the proposal to be more in line with the topic.

The point is that for some companies there's always going to be a good story to tell, if you really look for them.

For this event I've been able to get very good presentations from some big companies that initially were off of the track; I proactively reached out to them, but more importantly, their willingness to work with me to find a good topic was the key to get them accepted.

I had the opposite experience as well; some big companies didn't respond at all (even though they sent in a proposal); some wanted to stick with their original presentation no matter what; others indicated they didn't come up with anything interesting.

Ultimately, we had to reject many of these proposals.

Lesson learned

In the unlikely chance that a reviewer reaches out to you, it is usually a good sign that they want to work with you, and as the organizer, after you work on the content a bit can let them in. If you were not serious about your presentation, you might want to start to be from now on.

Not selected anyway

One thing that surprised me was the amount of proposals that were totally not in line with the conference's main topic, which was (I think/hope) well specified in the CFP form as well as on the website.

I am aware there are speakers that are submitting presentations in bulk no matter what the general topic is; I also found people submitting presentations with APIs and other buzzwords in the CFPs (see point n.1), but the meat was still not in line with the topic. They tried, I guess.

Some of these presentations, though, were really good. They were just not a good fit for this specific event and topic.

Please take time to read what the conference/event is about.

No matter how good your presentation is — if it is not in line with the conference topic you can put whatever buzzword you want, but you won't make it. And if you do for some reason, you're going to be marked as a False Positive (see the relative section above)

Timing is important

Since the inception of the conference (10 months ago), I had a clear idea of what the event should all be about. I had 4 main tracks in mind:

Stories from the transportation industry
APIs for B2B Applications
APIs for B2C Applications
APIs in Public Sector

As time passes and the CFPs are coming and coming, you will periodically review them and start sending notifications to your speakers, slowly filling the slots.

At a certain point, I realised that while I had some of the tracks pretty well covered, others were almost empty. For this reason, I had to change the strategy and give priority to empty tracks during the selection process.

Although the review process was almost the same, presentations in topic but not in the track that I wanted to fill were put on hold. Some of them were ultimately not selected even though their content was very good.

Lesson learned

Submit your proposal as soon as possible. The more empty spaces there are, the greater the chance you will be selected.

Security in your OpenAPI Specification

Vincenzo Chianese — Thu, 14 Feb 2019 12:34:25 +0000

Importance of Security during the API Design Phase

It is no longer a mystery, APIs are eating the world. There are a lot of companies today that are offering their APIs as the primary medium to interact with a system, with the User Interface merely as a byproduct, or, in any case, not the primary product being sold. Companies such as Stripe and Twilio led the vanguard of this movement in 2012.

Given this shift (which, by the way, is still happening), APIs have started to get more and more complicated, and so the API development process needed to evolve as well. APIs have become a product, therefore all the typical team members and methodologies have been employed to make that product successful.

The increase in relevance of API based products requires new tooling to support the people and methodologies used to implement the product. How would you design an API so that you can review and iterate on the design before building it for real? When designing a mobile application, you have tools to easily create a mockup to make sure everybody in your organization is on the same page before committing your resources on creating the real app.

For years, this wasn’t a thing for APIs. Until the OpenAPI Specification (and a bunch of other standards) came out.

In this article, we’ll review what OpenAPI can offer from a security standpoint, how you can embrace these features today, and showcase some companies that are doing really cool stuff in this space!

OpenAPI History: What, Why, When

OpenAPI — formerly known as Swagger — started out as a simple, open source specification for designing RESTful APIs in 2010 and, despite other API Specifications formats that came out during the following years (such as RAML and APIBlueprint), the Swagger project became the most popular one.

In 2015, the Swagger project was donated to the Linux Foundation and renamed to OpenAPI Specification, with Microsoft and IBM joining the foundation to help move the format forward. Their first release was OpenAPI 2.0 ,which is nothing more than the original Swagger format rebranded; and then a couple of years later, OpenAPI 3.0 was released with some important updates.

Today the OpenAPI initiative includes more than 10 companies that recognize the high value and importance of joining efforts to produce a standardized document to describe an API.

OpenAPI 2.0

Format overview

OpenAPI 2.0 is nothing more than the good old Swagger format, rebranded after the donation to the Linux Foundation.

OpenAPI 2.0 documents can be written both as a JSON or a YAML file, and they allow you to define how your exposed API looks like in terms of endpoints, accepted and returned payloads, media types, returned status codes, and servers where the API can be reached.

We aren’t going to look at all these parts here. Instead, we will focus on the security features that the specification offers.

Security Features of OpenAPI 2.0

OpenAPI 2.0 offers a dedicated section to declare the security features and requirements of your API and then use these where appropriate in your paths and operations.

api_key:
  type: apiKey
  name: api_key
  in: header
petstore_auth:
  type: oauth2
  authorizationUrl: http://swagger.io/api/oauth/dialog
  flow: implicit
  scopes:
    write:pets: modify pets in your account
    read:pets: read your pets

In this example we’ve declared 2 security definitions: the first one will be looking for an API Key in the specified header (api_key in this case), while the second one is declaring an implicit oAuth2 flow, requiring some particular scopes.

OpenAPI 2.0 supports another security definition type called basic , which is fundamentally the old plain HTTP Authentication format.

OpenAPI 2.0 does not have other built-in security definitions and it’s not possible to define custom ones without using vendor-provided extensions. While these three options cover a good chunk of real-world use cases, they might not be enough for special cases.

OpenAPI 3.0

Format overview and improvements over OpenAPI 2.0

OpenAPI 3.0 was released in 2017 with the intention of addressing some of the drawbacks and limitations version 2.0 was suffering from.

We are not going to focus on the OpenAPI 3.0 Specification and highlight all the differences between the two formats: there are plenty of links out on the internet detailing the changes. Instead, we are going to focus on the security changes that have been introduced in this new version of the specification.

Security Features of OpenAPI 3.0

In the same way OpenAPI 2.0 has a dedicated part of the document to declare security definitions, OpenAPI 3.0 has one too. The difference is that OpenAPI 3.0 has changed the terminology to "security schemes." The spec also standardized the way to declare all the parts of the spec that can be reused across multiple paths. While previously the shared components was something left to the common sense of the developers, now they are all grouped under the components key.

Moreover, OpenID Connect support has been added as well as the ability for a security definition to include multiple oAuth2 flows (which is a common functionality used around, today).

To give an idea of how the Security declaration have changed from OAS2.0 to OAS 3.0, this is the same example we mentioned before, wrote for OAS3:

securitySchemes:
  api_key:
    type: apiKey
    name: api_key
    in: header
  petstore_auth:
    type: oauth2
    flows:
      implicit:
        authorizationUrl: http://swagger.io/api/oauth/dialog
        scopes:
          write:pets: modify pets in your account
          read:pets: read your pets
      authorizationCode:
        authorizationUrl: http://swagger.io/api/oauth/dialog
        tokenUrl: https://swagger.io/api/oauth/token
        scopes:
          read:pets: Grants read access
          write:pets: Grants write access

You can see that now the oAuth2 definition supports multiple flows, defined by a new key in the security section.

Hands-On: Designing a Secure API

Let’s now go hands-on and try to create a document representing an API that’ll be exposed on the public internet. We will go through the design phase — as well as the implementation code and contract testing for it.

As most of the tooling around OpenAPI is still sticking with the 2.0 version, that is the version we are going to use. However, all the concepts we will be showing here today are still valid.

Create an OpenAPI Specification

OpenAPI specification documents are nothing more than YAML or JSON files. Although this is effectively lowering the barrier for new folks willing to write docs (everybody can write a JSON or a YAML document) this hides the complexity of the specification itself. This is where a visual designer or other tools can really be handy.

There are plenty of these on the market; namely Stoplight, Apiary, SwaggerHub. The decision of which one to choose is really up to you and your needs. In any case, all of them offer a free plan so you do not have to pay to get started.

We are going to write nothing more than a simple API that is a trimmed version of the well known example of the PetStore. This API is part of the official examples in both OAS2.0 and 3.0. I’ve chosen this one because of the simplicity of the content itself.

Write the code

Writing the code, in the world of APIs, is really the most trivial part. Building an API is more a conceptual work and a collaboration effort rather than committing stuff on your repository.

For this reason, we will stick to a very basic example, whose code is hosted on Glitch. For those who do not know it, Glitch is a free hosted service for NodeJS applications. It’s perfect for prototyping and shipping small applications in a very fast way.

Testing your API

Now what we really want to do is keep checking, as you’re developing the server, that the implementation we just wrote is in sync with the specification document we have in our repository. This is where the contract testing tools come into the play; they spin up an instance of your server and, by going through the OpenAPI document, they send HTTP requests to your servers which check the status codes as well as returned payload. If any of these do not match, it will make the CI/CD step fail, and the API will not be deployed. You can also see this as TDD development, where you initially have just an OpenAPI document with all tests failing because you do not have any code.

As you’re developing the API and adding more and more endpoint, you will start seeing more parts of the tests pass until you’re on green (which means you covered the entire API surface declared in the OpenAPI Document).

In the example we just did, I decided to go with Prism but Dredd is another one you can use. They both support OpenAPI 2.0, but unfortunately none of these support OpenAPI 3.0 (yet).

The setup for these tools is usually straightforward and does not require any particular effort.

The first thing we’re going to do is to download the Prism server on the machine:

curl -L https://github.com/stoplightio/prism/releases/download/v2.0.16/prism_linux_amd64 -o prism && chmod +x ./prism

Once done, we need to start our application server locally and then run Prism providing the listening URL to send the requests to and the OpenAPI document to use as a source of truth:


npm start
prism conduct ./sl/tests.scenarios.yml -e host=http://localhost:$PORT

Prism will now read the OpenAPI specification files, go through all the paths and start crafting HTTP Requests following the provided examples or creating payloads using the JSON Schema as a reference and send them to your application server. Once the response comes, Prism will verify the status code as well as that the response’s shape matches the declared one. If any of these do not match, the server will report and error and mark the test as a failure.

Note: these commands should be suitable in a CI/CD environment as well.

Integrate your OpenAPI Security Specification in Auth0

Now that we have an API whose design specification matches the implementation, it’s time to deploy it. Most likely we want to protect the API, and Auth0 is clearly one of the choices.

Not all the flows are currently supported by Auth0. Here’s a breakdown of the situation currently:

oAuth2: all the flows are supported and is the recommended way to deal with authentication/authorization in general. It is also possible to emit JSON based tokens instead of opaque ones, enabling other scenarios (such as passing the token around web apps)
openIdConnect: supported by Auth0 with all the standard claims
basic/http: supported as a particular type of oAuth 2.0 flow. Any client that has a client password might send this using the HTTP Basic Authentication Scheme. To be fair, this is rather a way to send the client password rather than full support for the scheme; as you won’t get any prompt on your browser while trying to hit the target path
apiKey: Not supported. This is not formalized in any standard per se — but it is a common way to send a preshared credential on the wire. The nearest authentication method that resembles the presence of an API Key is to use a Client Credential Grant Request.

You can see that, as long you’re using oAuth2.0 or openIdConnect, there’s nothing to worry about. You might get in trouble in case you want to stick with some "older" methods, which can be emulated to a certain degree.

Runtime integration

At this stage, we have a backend server implementing our API and an Identity Provider (in this case Auth0) that is storing all our users and providing the necessary infrastructure to authenticate and authorize them in our platform. We are missing the glue between the two pieces: how can an user willing to authenticate in my application get redirected to Auth0’s services?

You can do that in your application code, but in that case it’s no more declarative and a change in your OpenAPI document regarding the security requirements will require a change in your code too.

There’s a possibility to automate this stage with the help of an API Gateway. Given this piece of software is usually configured in an declarative manner and the fact that most of the time your OpenAPI document matches the exposed API, there’s almost a perfect match between the spec and the API Gateway configuration. Unfortunately, as of today, none of the Gateways on the market have reached such level of integration.

For people that are interested in this topic, I gave a presentation last December whose video is published online

Conclusion

We have gone through the security features that are part of OpenAPI 2.0 and 3.0 specifications, and they can help the API users and developers clearly set expectations when consuming and building an API. Then we went a little bit into the API lifecycle and, as you probably noticed, it’s more about communication, being on the same page, and about what you want to expose rather than simply writing the code. That’s just a single step in the whole process. In the end, we explored the opportunities of a runtime integration with an Identity Provider, which is not there yet.

DEV Community: Vincenzo Chianese

Things I discovered while writing Clojure

Succinct JavaScript?

Break functions apart

From 0 to Reader

Relocating during the COVID-19

Prague, December 2014

Madrid, September 2017

Austin, March 2020

Flight schedule: gone

The loop

Building a credit score

The skin rashes

What about now?

Frequently Asked Questions

What’s the state of the functional rewrite?

What do your coworkers think about it

How much time did it take to get this far? Was it worth it at all?

Why not using Haskell/F#/Clojure/WhatEverFancyLanguage?

Do you think FP makes always sense?

Are you seriously betting all this on an unknown library called fp-ts?

What about the performances?

Thanks!

The road to the return of the investment

First Expansion: Router

Expand fs-ts in Prism's router #402

Return Of Investment: Validation

When It Went Wrong: Security Checks

What are the lessons learned?

Introducing in the company

Finding an introduction point

Logging in Prism

Getting the PR merged

Logging Prism's decisions. #323

TL; DR

Long version

Outstanding points

Finding the right abstraction

A validator in TypeScript

First impressions

Giulio CantiFollow

Either 101

Forewords and domain model

What Is Prism

stoplightio / prism

Turn any OpenAPI2/3 and Postman Collection file into an API server with mocking, transformations and validations.

Prism Peculiarities

100% TypeScript

A Lot Of Custom Software

Custom Negotiator

Input, output and security validation

Prism Request Flow

Prism User

A reviewer's perspective on getting your conference proposal accepted

Events get a lot of proposals

Problem: it’s too time consuming to evaluate every proposal in detail

Lesson learned: Title and Abstract are the most important filters

Problem: titles and abstract might be misleading

Lesson learned (for the reviewers)

Lesson learned (for the speakers)

Work with the speakers

Lesson learned

Not selected anyway

Please take time to read what the conference/event is about.

Sponsors

Lesson learned (for the reviewers)

Lesson learned (for the sponsors)

Timing is important

Lesson learned

Security in your OpenAPI Specification

Importance of Security during the API Design Phase

OpenAPI History: What, Why, When

OpenAPI 2.0

Format overview

Security Features of OpenAPI 2.0

OpenAPI 3.0

Format overview and improvements over OpenAPI 2.0

Security Features of OpenAPI 3.0

Hands-On: Designing a Secure API

Create an OpenAPI Specification

Are you seriously betting all this on an unknown library called `fp-ts`?