DEV Community: NoRedInk

Haskell for the Elm Enthusiast

Jasper Woudenberg — Tue, 03 Aug 2021 15:40:01 +0000

This post was co-authored by Michael Glass, Stöffel, and myself. It first first appeared on the NoRedInk blog.

Many years ago NRI adopted Elm as a frontend language. We started small with a disposable proof of concept, and as the engineering team increasingly was bought into Elm being a much better developer experience than JavaScript more and more of our frontend development happened in Elm. Today almost all of our frontend is written in Elm.

Meanwhile, on the backend, we use Ruby on Rails. Rails has served us well and has supported amazing growth of our website, both in terms of the features it supports, and the number of students and teachers who use it. But we’ve come to miss some of the tools that make us so productive in Elm: Tools like custom types for modeling data, or the type checker and its helpful error messages, or the ease of writing (fast) tests.

A couple of years ago we started looking into Haskell as an alternative backend language that could bring to our backend some of the benefits we experience writing Elm in the frontend. Today some key parts of our backend code are written in Haskell. Over the years we’ve developed our style of writing Haskell, which can be described as very Elm-like (it’s also still changing!).

🌳 Why be Like Elm?

Elm is a small language with great error messages, great documentation, and a great community. Together these make Elm one of the nicest programming languages to learn. Participants in an ElmBridge event will go from knowing nothing of the language to writing a real application using Elm in 5 hours.

We have a huge amount of Elm code at NoRedInk, and it supports some pretty tricky UI work. Elm scales well to a growing and increasingly complicated codebase. The compiler stays fast and we don’t lose confidence in our ability to make changes to our code. You can learn more about our Elm story here.

📦 Unboxing Haskell

Haskell shares a lot of the language features we like in Elm: Custom types to help us model our data. Pure functions and explicit side effects. Writing code without runtime exceptions (mostly).

When it comes to ease of learning, Haskell makes different trade-offs than Elm. The language is much bigger, especially when including the many optional language features that can be enabled. It’s entirely up to you whether you want to use these features in your code, but you’ll need to know about many of them if you want to make use of Haskell’s packages, documentation, and how-tos. Haskell’s compiler errors typically aren’t as helpful as Elm’s are. Finally, we’ve read many Haskell books and blog posts, but haven’t found anything getting us from knowing no Haskell to writing a real application in it that’s anywhere near as small and effective as the Elm Guide.

🏟️ When in Rome, Act Like a Babylonian

Many of the niceties we’re used to in Elm we get in Haskell too. But Haskell has many additional features, and each one we use adds to the list of things that an Elm programmer will need to learn. So instead we took a path that many in the Haskell community took before us: limit ourselves to a subset of the language.

There are many styles of writing Haskell, each with its own trade-offs. Examples include Protolude, RIO, the lens ecosystem, and many more. Our approach differs in being strongly inspired by Elm. So what does our Elm-inspired style of writing Haskell look like?

🍇 Low hanging fruit: the Elm standard library

Our earliest effort in making our Haskell code more Elm-like was porting the Elm standard library to Haskell. We’ve open-sourced this port as a library named nri-prelude. It contains Haskell counterparts of the Elm modules for working with Strings, Lists, Dicts, and more.

nri-prelude also includes a port of elm-test. It provides everything you need for writing unit tests and basic property tests.

Finally, it includes a GHC plugin that makes it so Haskell’s default Prelude (basically its standard library) behaves like Elm’s defaults. For example, it adds implicit qualified imports of some modules like List, similar to what Elm does.

🎚️ Effects and the Absence of The Elm Architecture

Elm is opinionated in supporting a single architecture for frontend applications, fittingly called The Elm Architecture. One of its nice qualities is that it forces a separation of application logic (all those conditionals and loops) and effects (things like talking to a database or getting the current time). We love using The Elm Architecture writing frontend applications, but don’t see a way to apply it 1:1 to backend development. In the F# community, they use the Elm Architecture for some backend features (see: When to use Elmish Bridge), but it’s not generally applicable. We’d still like to encourage that separation between application logic and effects though, having seen some of the effects of losing that distinction in our backend code. Read our other post Pufferfish, please scale the site! if you want to read more about this.

Out of many options we’re currently using the handle pattern for managing effects. For each type of effect, we create a Handler type (we added the extra r in a typo way back and it has stuck around. Sorry). We use this pattern across our libraries for talking to outside systems: nri-postgresql, nri-http, nri-redis, and nri-kafka.

Without The Elm Architecture, we depend heavily on chaining permutations through a stateful Task type. This feels similar to imperative coding: First, do A, then B, then C. Hopefully, when we’re later on in our Haskell journey, we’ll discover a nice architecture to simplify our backend code.

🚚 Bringing Elm Values to Haskell

One way in which Haskell is different from both Elm and Rails is that it is not particularly opinionated. Often the Haskell ecosystem offers multiple different ways to do one particular thing. So whether it’s writing an http server, logging, or talking with a database, the first time we do any of these things we’ll need to decide how.

When adopting a Haskell feature or library, we care about

smallness, e.g. introduce new concepts only when necessary
how “magical” is it? E.g. How surprising is it?
How easy is it to learn?
how easy is it to use?
how comprehensible is the documentation?
explicitness over *terseness (*but terseness isn’t implicitly bad).
consistency & predictability
“safety” (no runtime exceptions).

Sometimes the Haskell ecosystem provides an option that fits our Elm values, like with the handle pattern, and so we go with it. Other times a library has different values, and then the choice not to use it is easy as well. An example of this is lens/prism ecosystem, which allows one to write super succinct code, but is almost a language onto itself that one has to learn first.

The hardest decisions are the ones where an approach protects us against making mistakes in some way (which we like) but requires familiarity with more language features to use (which we prefer to avoid).

To help us make better decisions, we often try it both ways. That is, we’re willing to build a piece of software with & without a complex language feature to ensure the cost of the complexity is worth the benefit that the feature brings us.

Another approach we take is making decisions locally. A single team might evaluate a new feature, and then demo it and share it with other teams after they have a good sense the feature is worth it. Remember: a super-power of Haskell is easy refactorability. Unlike our ruby code, going through and doing major re-writes in our Haskell codebase is often an hours-or-days-long (rather than weeks-or-months-long) endeavor. Adopting two different patterns simultaneously has a relatively small cost!

Case studies in feature adoption:

🐘 Type-Check All Elephants

One example where our approach is Elm-like in some ways but not in others is how we talk to the database. We’re using a GHC feature called quasiquoting for this, which allow us to embed SQL query strings directly into our Haskell code, like this:

{-# LANGUAGE QuasiQuotes #-}

module Animals (listAll) where

import Postgres (query, sql)

listAll :: Postgres.Handler -> Task Text (List (Text, Text))
listAll postgres =
  query postgres [sql|SELECT species, genus FROM animals|]

A library called postgresql-typed can test these queries against a real Postgres database and show us an error at compile time if the query doesn’t fit the data. Such a compile-time error might happen if a table or column we reference in a query doesn’t exist in the database. This way we use static checks to eliminate a whole class of potential app/database compatibility problems!

The downside is that writing code like this requires everyone working with it to learn a bit about quasi quotes, and what return type to expect for different kinds of queries. That said, using some kind of querying library instead has a learning curve too, and query libraries tend to be pretty big to support all the different kinds of queries that can be made.

🔣 So Many Webserver Options

Another example where we traded additional safety against language complexity is in our choice of webserver library. We went with servant here, a library that lets you express REST APIs using types, like this:

import Servant

data Routes route = Routes
  { listTodos ::
      route
        :- "todos"
        :> Get '\[JSON\] [Todo],
    updateTodo ::
      route
        :- "todos"
        :> Capture "id" Int
        :> ReqBody '[JSON] Todo
        :> Put '[JSON] NoContent,
    deleteTodo ::
      route
        :- "todos"
        :> Capture "id" Int
        :> Delete '[JSON] NoContent
  }
  deriving (Generic)

Servant is a big library that makes use of a lot of type-level programming techniques, which are pretty uncommon in Elm, so there’s a steep learning cost associated with understanding how the type magic works. Using it without a deep understanding is reasonably straightforward.

The benefits gained from using Servant outweigh the cost of expanded complexity. Based on a type like the one in the example above, the servant ecosystem can generate functions in other languages like Elm or Ruby. Using these functions means we can save time with backend-to-frontend or service-to-service communication. If some Haskell type changes in a backward-incompatible fashion we will generate new Elm code, and this might introduce a compiler error on the Elm side.

So for now we’re using servant! It’s important to note that what we want is compile-time server/client compatibility checking, and that’s why we swallow Servant’s complexity. If we could get the same benefit without the type-level programming demonstrated above, we would prefer that. Hopefully, in the future, another library will offer the same benefits from a more Elm-like API.

😻 Like what you see?

We're running the libraries discussed above in production. Our most-used Haskell application receives hundreds of thousands of requests per minute without issue and produces hardly any errors.

Code can be found at NoRedInk/haskell-libraries. Libraries have been published to hackage and stackage. We'd love to know what you think!

☄️ Pufferfish, please scale the site!

Jasper Woudenberg — Thu, 22 Jul 2021 07:43:08 +0000

This post was co-authored by Michael Glass, Stöffel, and myself. It first first appeared on the NoRedInk blog.

We created Team Pufferfish about a year ago with a specific goal: to avert the MySQL apocalypse! The MySQL apocalypse would occur when so many students would work on quizzes simultaneously that even the largest MySQL database AWS has on offer would not be able to cope with the load, bringing the site to a halt.

A little over a year ago, we forecasted our growth and load-tested MySQL to find out how much wiggle room we had. In the worst case (because we dislike apocalypses), or in the best case (because we like growing), we would have about a year’s time. This meant we needed to get going!

Looking back on our work now, the most important lesson we learned was the importance of timely and precise feedback at every step of the way. At times we built short-lived tooling and process to support a particular step forward. This made us so much faster in the long run.

🏔 Climbing the Legacy Code Mountain

Clear from the start, Team Pufferfish would need to make some pretty fundamental changes to the Quiz Engine, the component responsible for most of the MySQL load. Somehow the Quiz Engine would need to significantly reduce its load on MySQL.

Most of NoRedInk runs on a Rails monolith, including the Quiz Engine. The Quiz Engine is big! It’s got lots of features! It supports our teachers & students to do lots of great work together! Yay!

But the Quiz Engine has some problems, too. A mix of complexity and performance-sensitivity has made engineers afraid to touch it. Previous attempts at big structural change in the Quiz Engine failed and had to be rolled back. If Pufferfish was going make significant structural changes, we would need to ensure our ability to be productive in the Quiz Engine codebase. Thinking we could just do it without a new approach would be foolhardy.

⚡ ️The Vengeful God of Tests

We have mixed feelings about our test suite. It’s nice that it covers a lot of code. Less nice is that we don’t really know what each test is intended to check. These tests have evolved into complex bits of code by themselves with a lot of supporting logic, and in many cases, tight coupling to the implementation. Diving deep into some of these tests has uncovered tests no longer covering any production logic at all. The test suite is large and we didn’t have time to dive deep into each test, but we were also reluctant to delete test cases without being sure they weren’t adding value.

Our relationship with the Quiz Engine test suite was and still is a bit like one might have with an angry Greek god. We’re continuously investing effort to keep it happy (i.e. green), but we don’t always understand what we’re doing or why. Please don’t spoil our harvest and protect us from (production) fires, oh mighty RSpec!

The ultimate goal wasn’t to change Quiz Engine functionality, but rather to reduce its load on MySQL. This is the perfect scenario for tests to help us! The test suite we want is:

fast
comprehensive, and
not dependent on implementation
includes performance testing

Unfortunately, that’s not the hand we were given:

The suite takes about 30 minutes to run in CI and even longer locally.
Our QA team finds bugs that sneaked past CI in PRs with Quiz Engine changes relatively frequently.
Many tests ensure that specific queries are performed in a specific order. Considering we might replace MySQL wholesale, these tests provide little value.
And because a lot of Quiz Engine code is extremely performance-sensitive, there’s an increased risk of performance regressions only surfacing with real production load.

Fighting with our tests meant that even small changes would take hours to verify in tests, and then, because of unforeseen regressions not covered by the tests, take multiple attempts to fix, resulting in multiple-day roll-outs for small changes.

Our clock is ticking! We needed to iterate faster than that if we were going to avert the apocalypse.

🐶 I have no idea what I’m doing 🧪

Reading complicated legacy Rails code often raises questions that take surprising amounts of effort to answer.

Is this method dead code? If not, who is calling this?
Are we ever entering this conditional? When?
Is this function talking to the database?
Is this function intentionally talking to the database?
Is this function only reading from the database or also writing to it?

It isn’t even clear what code was running. There are a few features of Ruby (and Rails) which optimize for writing code over reading it. We did our best to unwrap this type of code:

Rails provides devs the ability to wrap functionality in hooks. before_ and after_ hooks let devs write setup and tear-down code once, then forget it. However, the existence of these hooks means calling a method might also evaluate code defined in a different file, and you won’t know about it unless you explicitly look for it. Hard to read!

Complicating things further is Ruby’s dynamic dispatch based on subclassing and polymorphic associations. Which load_students am I calling? The one for Quiz or the one for Practice? They each implement the Assignment interface but have pretty different behavior! And: they each have their own set of hooks🤦. Maybe it’s something completely different!

And then there’s ActiveRecord. ActiveRecord makes it easy to write queries — a little too easy. It doesn’t make it easy to know where queries are happening. It’s ergonomic that we can tell ActiveRecord what we need, and let it figure how to fetch the data. It’s less nice when you’re trying to find out where in the code your queries are happening and the answer to that question is, “absolutely anywhere”. We want to know exactly what queries are happening on these code paths. ActiveRecord doesn’t help.

🧵A rich history

A final factor that makes working in Quiz Engine code daunting is the sheer size of the beast. The Quiz Engine has grown organically over many years, so there’s a lot of functionality to be aware of.

Because the Quiz Engine itself has been hard to change for a while, APIs defined between bits of Quiz Engine code often haven’t evolved to match our latest understanding. This means understanding the Quiz Engine code requires not just understanding what it does today, but also how we thought about it in the past, and what (partial) attempts were made to change it. This increases the sum of Quiz Engine knowledge even further.

For example, we might try to refactor a bit of code, leading to tests failing. But is this conditional branch ever reached in production? 🤷

Enough complaining. What did we do about it?

We knew this was going to be a huge project, and huge projects, in the best case, are shipped late, and in the average case don’t ever ship. The only way we were going to have confidence that our work would ever see the light of day was by doing the riskiest, hardest, scariest stuff first. That way, if one approach wasn’t going to work, we would find out about it sooner and could try something new before we’d over-invested in a direction.

So: where is the risk? What’s the scariest problem we have to solve? History dictates: The more we change the legacy system, the more likely we’re going to cause regressions.

So our first task: cut away the part of the Quiz Engine that performs database queries and port this logic to a separate service. Henceforth when Rails needs to read or change Quiz Engine data, it will talk to the new service instead of going to the database directly.

Once the legacy-code risk has been minimized, we would be able to focus on the (still challenging) task of changing where we store Quiz Engine data from single-database MySQL to something horizontally scalable.

⛏️ Phase 1: Extracting queries from Rails

🔪 Finding out where to cut

Before extracting Quiz Engine MySQL queries from our Rails service, we first needed to know where those queries were being made. As we discussed above this wasn’t obvious from reading the code.

To find the MySQL queries themself, we built some tooling: we monkey-patched ActiveRecord to warn whenever an unknown read or write was made against one of the tables containing Quiz Engine data. We ran our monkey-patched code first in CI and later in production, letting the warnings tell us where those queries were happening. Using this information we decorated our code by marking all the reads and writes. Once code was decorated, it would no longer emit warnings. As soon as all the writes & reads were decorated, we changed our monkey-patch to not just warn but fail when making a query against one of those tables, to ensure we wouldn’t accidentally introduce new queries touching Quiz Engine data.

🚛 Offloading logic: Our first approach

Now we knew where to cut, we decided our place of greatest risk was moving a single MySQL query out of our rails app. If we could move a single query, we could move all of them. There was one rub: if we did move all queries to our new app, we would add a lot of network latency. because of the number of round trips needed for a single request. Now we have a constraint: Move a single query into a new service, but with very little latency.

How did we reduce latency?

Get rid of network latency by getting rid of the network — we hosted the service in the same hardware as our Rails app.
Get rid of protocol latency by using a dead-simple protocol: socket communication.

We ended up building a socket server in Haskell that took data requests from Rails, and transformed them into a series of MySQL queries, which rails would use to fetch the data itself.

🛸 Leaving the Mothership: Fewer Round Trips

Although co-locating our service with rails got us off the ground, it required significant duct tape. We had invested a lot of work building nice deployment systems for HTTP services and we didn’t want to re-invent that tooling for socket-based side-car apps. The thing that was preventing the migration was having too many round-trip requests to the Rails app. How could we reduce the number of round trips?

As we moved MySQL query generation to our new service, we started to see this pattern in our routes:

MySQL Read some data       ┐
Ruby  Do some processing   │ candidate 1 for
MySQL Read some more data  ┘ extraction
Ruby  More processing
MySQL Write some data      ┐
Ruby  Processing again!    │ candidate 2 for
MySQL Write more data      ┘ extraction

To reduce latency, we’d have to bundle reads and writes: In addition to porting reads & writes to the new service, we’d have to port the ruby logic between reads and writes, which would be a lot of work.

What if instead, we could change the order of operations and make it look like this?

MySQL Read some data       ┐ candidate 1 for
MySQL Read some more data  ┘ extraction
Ruby  Do some processing
Ruby  More processing
Ruby  Processing again!
MySQL Write some data      ┐ candidate 2 for
MySQL Write more data      ┘ extraction

Then we’d be able to extract batches of queries to Haskell and leave the logic behind in Rails.

One concern we had with changing the order of operations like this was the possibility of a request handler first writing some data to the database, then reading it back again later. Changing the order of read and write queries would result in such code failing. However, since we now had a complete and accurate picture of all the queries the Rails code was making, we knew (luckily!) we didn’t need to worry about this.

Another concern was the risk of a large refactor like this resulting in regressions causing long feedback cycles and breaking the Quiz Engine. To avoid this we tried to keep our refactors as dumb as possible: Specifically: we mostly did a lot of inlining. We would start with something like this

class QuizzesControllller < ActionController
  def show
    quiz = load_quiz! # here are queries sometimes
    quiz_type = which_quiz(quiz) # and here other times
  end
end

and we would aggressively inline functions to surface where and why we were querying

class QuizzesControllller < ActionController
  def show
    quiz = Quiz.find(quiz_id_param)
    quiz_type =
      if quiz.for_credit?
        :for_credit
      else
        load_practice_quiz_type
      end
  end
end

and again, and again

class QuizzesControllller < ActionController
  def show
    quiz = Quiz.find(quiz_id_param)
    quiz_type =
      if quiz.for_credit?
        :for_credit
      else
        how_much_fun = QuizForFun.find(quiz_id_param)
        if how_much_fun > 9000
          :super_saiyan
        else
          load_sub_syan_fun_type # TODO: inline me
        end
      end
  end
end

These are refactors with a relatively small chance of changing behavior or causing regressions.

Once the query was at the top level of the code it became clear when we needed data, and that understanding allowed us to push those queries to happen first.

e.g. from above, we could easily push the previously obscured QuizForFun query to the beginning:

class QuizzesControllller < ActionController
  def show
    quiz = Quiz.find(quiz_id_param)
    how_much_fun =
      if quiz.for_credit?
        nil
      else
        QuizForFun.find(quiz_id_param)
      end
    quiz_type = if quiz.for_credit?
        :for_credit
      elsif how_much_fun > 9000
        :super_saiyan
      else
        load_sub_syan_fun_type # TODO: inline me
      end
  end
end

You might expect our bout of inlining to introduce a ton of duplication in our code, but in practice, it surfaced a lot of dead code and made it clearer what the functions we left behind were doing. That wasn’t what we set out to do, but still, nice!

👛 Phase 2: Changing the Quiz Engine datastore

At this point all interactions with the Quiz Engine datastore were going through this new Quiz Engine service. Excellent! This means for the second part of this project, the part where we were actually going to avert the MySQL apocalypse, we wouldn’t need to worry about our legacy Rails code.

To facilitate easy refactoring, we built this new service in Haskell. The effect was immediately noticeable. Like an embargo had been lifted, from this point forward we saw a constant trickle of small productive refactors get mixed in the work we were doing, slowly reshaping types to reflect our latest understanding. Changes we wouldn’t have made on the Rails side unless we’d have set aside months of dedicated time. Haskell is a great tool to use to manage complexity!

The centerpiece of this phase was the architectural change we were planning to make: switching from MySQL to a horizontally scalable storage solution. But honestly, figuring the architecture details here wasn’t the most interesting or challenging portion of the work, so we’re just putting that aside for now. Maybe we’ll return to it in a future blog post (sneak peek: we ended up using Redis and Kafka). Like in step 1, the biggest question we had to solve was “how are we going to make it safe to move forward quickly?”

One challenge was that we had left most of our test suite behind in Rails in phase one, so we were not doing too well on that front. We added Haskell test coverage of course, including many golden result tests which are worth a post on their own. Together with our QA team we also invested in our Cypress integration test suite which runs tests from the browser, thus integration-testing the combination of our Rails and Haskell code.

Our most useful tool in making safe changes in this phase however was our production traffic. We started building up what was effectively a parallel Haskell service talking to Redis next to the existing one talking to MySQL. Both received production load from the start, but until the very end of the project only the MySQL code paths’ response values were used. When the Redis code path didn’t match the MySQL, we’d log a bug. Using these bug reports, we slowly massaged the Redis code path to return identical data to MySQL.

Because we weren’t relying on the output of the Redis code path in production, we could deploy changes to it many times a day, without fear of breaking the site for students or teachers. These deploys provided frequent and fast feedback. Deploying frequently was made possible by the Haskell Quiz Engine code living in its own service, which meant deploys contained only changes by our team, without work from other teams with a different risk profile.

🥁 So, did it work?

It’s been about a month since we’ve switched entirely to the new architecture and it’s been humming along happily. By the time we did the official switch-over to the new datastore it had been running at full-load (but with bugs) for a couple of months already. Still, we were standing ready with buckets of water in case we overlooked something. Our anxiety was in vain: the roll-out was a non-event.

Architecture, plans, goals, were all important to making this a success. Still, we think the thing most crucial to our success was continuously improving our feedback loops. Fast feedback (lots of deploys), accurate feedback (knowing all the MySQL queries Rails is making), detailed feedback (lots of context in error reports), high signal/noise ratio (removing errors we were not planning to act on), lots of coverage (many students doing quizzes). Getting this feedback required us to constantly tweak and create tooling and new processes. But even if these processes were sometimes short-lived, they've never been an overhead, allowing us to move so much faster.

Type-Safe MySQL Queries via Postgres

Christoph Hermann — Tue, 05 Nov 2019 08:19:05 +0000

Type-Safe MySQL Queries via Postgres

My team and I are working on NoRedInk's transition from Ruby on Rails and MySQL to Haskell and Postgres. A major benefit of our new stack (Postgres/Haskell/Elm) is type-safety between the boundaries (db to backend, backend to frontend). To ensure our code is type-safe between Haskell and Elm we use servant. Servant allows us to generate types and functions for Elm based on our APIs. To ensure type-safety between Postgres and Haskell we use postgresql-typed. This means that when we change our database schema, our Haskell code won't compile if there is a mismatch, similarly to how our Elm code won't compile if there is a mismatch with our backend endpoints. We're going to focus on type-safety between our databases and Haskell in this post and on how we introduced MySQL into this. But first, let's briefly look at how we can check out Postgres queries at compile-time.

Postgres-Typed

todos :: Int -> PGConnection -> IO [Todo]
todos userId postgresConn = do
  rows <-
    pgQuery
      postgresConn
      [pgSQL|!
        SELECT id, description, completed
        FROM todos
        WHERE user_id = ${userId}
        ORDER BY id ASC
      |]
  pure (fmap todoFromRow rows)

We use Template Haskell to write SQL queries instead of a DSL, allowing our engineers to use their existing knowledge of SQL instead of having to learn some library. pgSQL is a QuasiQuoter it creates a value of type PGQuery which gets executed by pgQuery. The quasi quoted query gets verified against the database schema and executed (ignoring results and without arguments) at compile-time. This is all we need to know from postgresql-typed for this post, but I recommend checking out the docs and looking at their example.

Moving to Haskell and Postgres

Our Rails application uses a MySQL database. Gradually moving functionality/new features to Haskell often means that we need access to data that isn't yet moved. We solved this initially by creating an endpoints on our Rails application and requested the data via http. This proved to be toilsome and complicated development and deploys, and therefore made it harder for teams to commit to building things in Haskell. In order to remove this hurdle, we started to think about reading data directly from our MySQL database. We really liked postgresql-typed, because it gave us the ability to write actual SQL queries and type-safety between the database and Haskell. Unfortunatlly, there isn’t a mysql-typed, but we found an interesting solution provided by Postgres itself; foreign-data-wrappers.

Foreign Data Wrappers

Foreign Data Wrappers (short fdw) offer a way to manage remote data directly within your Postgres database. This allows us to use postgresql-typed to access the MySQL data via Postgres.

CREATE SERVER mysql-schema-name
  FOREIGN DATA WRAPPER mysql_fdw
  OPTIONS (host '${host}', port '${port}');

-- Mapping for the application user, so it can perform queries.
CREATE USER MAPPING FOR ${username}
  SERVER mysql-schema-name
  OPTIONS (username '${username}', password '${passphrase}');

CREATE SCHEMA mysql-schema-name AUTHORIZATION ${username};

IMPORT FOREIGN SCHEMA ${dbname}
  FROM SERVER mysql-schema-name
  INTO mysql-schema-name;

Unfortunately, we quickly ran into problems when converting http requests to the Rails application with queries directly to MySQL. Fdw has some limitations that were blockers for use. The schema imported as a fdw is imported without any constraints other than NOT NULL. An even bigger problem was that some SQL constructs weren’t pushed down to MySQL. As an example the fdw wouldn’t forward LIMIT clauses and therefore grab all rows of a table, send them over the wire, and then apply the LIMIT in Postgres. This obviously has huge performance implications and meant that we needed to find a different solution without suffering type-safety.

Type-Safe MySQL Queries

We had three options; either find a library that would give us the guarantees we were used to from postgresql-typed for MySQL, building our own library or somehow abusing postgresql-typed. Building our own library wasn't an option because that would have exceeded the scope of this project, and we couldn't find a library that met our requirements. Fortunately, postgresql-typed allows us to access getQueryString to get the raw query string from the quasi-quoted query. This we can then execute with mysql-simple instead of postgresql-typed. This allows us to check MySQL and Postgres queries at compile time using a fdw, but connecting directly to Postgres and MySQL at runtime. Finally we can write queries to our MySQL database using the same API as for Postgres and having full type-safety between the database and Haskell.

cats :: Database.MySQL.Simple.Connection -> IO [Cat]
cats mySQLConnection = do
  rows <-
    Database.MySQL.Simple.query_
      mySQLConnection $ -- direct connection to MySQL
      remove "mysql-schema-name." $
      -- ^ The schemaname (fdw) only exists during compile time.
      getQueryString unknownPGTypeEnv
      [pgSql|!
        SELECT id, description, color
        FROM mysql-schema-name.cats
        ORDER BY id ASC
      |]
  pure (fmap catFromRow rows)

I’ve inlined the functions from postgresql-typed and mysql-simple to keep the code examples simple. We actually have two modules MySQL and Postgres that abstract the database specific functions away allowing us to use the exact same API for both databases.

Caveats

We need to prefix tables with the schema name that we used to import the fdw (see mysql-schema-name.cats in the example above). Without the prefix the compilation won't succeed. The problem is that we don't have a schema with this name at runtime. We can simply work around this by running Text.replace "mysql-schema-name." "" sqlString before executing the query. Queries need to use standard SQL features and shouldn't rely on MySQL specific features. This is actually kind of a feature, because it forces us to write queries in a way that simplifies converting them to Postgres later.

Conclusion

While this is a nice solution for us, I don't really recommend you to use MySQL in this way. This solution allows us to simplify the transition between Rails/MySQL and Haskell/Postgres. I would love for library like postgresql-typed to exist for MySQL, but it wasn't feasable to build such a thing. Honestly, I don't even know if it would be possible. I hope this post showed you how to create a nice experience for transitioning to a different technology.
Thanks for reading ❤️

Special thanks to everyone involved in building this:

Drag & Drop without Draggables & Dropzones

Jasper Woudenberg — Fri, 02 Aug 2019 16:21:20 +0000

Why is building drag & drop UIs so hard? They've been around for a while, so we would be forgiven for thinking they're a solved problem. Certainly there are high quality libraries to help us build drag & drop UIs, and these days we even have an official HTML5 drag & drop API! What's going on here?

I've come to the conclusion a big part of the problem is our choice of tools. Draggables and dropzones are often the wrong abstraction, and choosing different tools can simplify construction of our drag & drop UIs massively.

That's a pretty sweeping statement for a broad term like 'drag & drop', so let's look at three examples of drag & drop UIs built without draggables or dropzones. These examples come from real projects, but I've stripped them down to their drag & drop essentials for this post. I'll include what I believe to be the most relevant code snippets in this post, and provide links to the full source code.

Let's start with a quick refresher on draggables and dropzones before we get to the examples.

A Refresher on Draggables & Dropzones

A draggable is a UI element that follows the cursor when the user presses down on it. A dropzone is a UI element that gets an event when we release a draggable over it.

For an example lets look at the application Trello. Trello allows us to organize cards in lists. We could make the cards draggables and the lists dropzones. That way we'd get an event every time a card gets dropped on a list, which we could use to update our app's state with the new location of the card.

On closer inspection though things aren't as clear cut. For example, in the real Trello application it's not necessary to drop a card on top of a list. We can drop a card on some empty space and it will move into the list nearest to where we dropped it. That's much nicer for the user, but it's not clear how to create this behavior using draggables and dropzones.

Our first example, A Timeslot Selector

Our first example is a UI for selecting a time slot in a day. This UI represents a day as a horizontal bar, in which we can select a slot using drag & drop.

I'd like to start these examples by imagining what a draggables and dropzones implementation might look like. In this first example it's unclear what our draggables and dropzones even are. We could make the time slider the draggable, because clicking anywhere in it should start the drag operation even if we don't drag the time slider itself. The screen as a whole might serve as a dropzone, because the user should be able to release the cursor anywhere to finish the drag. Already this approach feels pretty hacky, and we haven't even written any code yet!

Let's start from scratch without assuming draggables or dropzones. As we regularly do, we'll begin building our Elm application by designing a Model. Our application needs to store a time slot which is nothing more than a start and end hour.

-- In this example a time slot always covers whole hours,
-- but we could make this minutes or seconds if we wanted.
type alias Model =
    { selectionStart : Hour, selectionEnd : Hour }

type alias Hour =
    Int

Now we turn to implementing the drag & drop behavior, without draggables and dropzones. When the user presses down we store the hour the cursor is over as the selectionStart field of the timeslot. Then as the cursor moves we will update the selectionEnd field of the timeslot, until the user releases.

At any point we will need to know which hour the cursor is over. We can calculate this if we know the position of the cursor and the position and dimensions of the slider on the screen. Lets be optimistic and assume we just get that information on our drag events. If so we can design our Msg type like this:

type alias Msg =
    { event : DragEvent
    , cursor : Coords
    , sliderPosition : Rect
    }


type DragEvent
    = Start
      -- We don't need to do anything special on a Stop event, so we can treat
      -- it the same as a Move event.
    | MoveOrStop


type alias Coords =
    { x : Float, y : Float }


type alias Rect =
    { x : Float, y : Float, width : Float, height : Float }

The information in this Msg is enough to calculate the hour the cursor is over.

cursorAtHour : Msg -> Hour
cursorAtHour { cursor, sliderPosition } =
    let
        dx =
            cursor.x - sliderPosition.x

        atMost =
            min

        atLeast =
            max
    in
    (24 * (dx / sliderPosition.width))
        |> floor
        -- Ensure we get a number between 0 and 23, even if the cursor moves to
        -- the left or right of the slider.
        |> atMost 23
        |> atLeast 0

All that's left to do is use cursorAtHour in our update function. When we get a Start event we use it to update the selectionStart field in the model, and when we get a MoveOrStop event the selectionEnd field.

update : Msg -> Model -> Model
update msg model =
    let
        hour =
            cursorAtHour msg
    in
    case msg.event of
        Start ->
            if coordsInRect msg.cursor msg.sliderPosition then
                { selectionStart = hour
                , selectionEnd = hour
                }

            else
                model

        MoveOrStop ->
            { model
                | selectionEnd = hour
            }


coordsInRect : Coords -> Rect -> Bool
coordsInRect =
    Debug.todo "Implementation omitted for brevity."

And we have an app! Well almost, we've not discussed where these Msgs will be coming from. There's a bit of JavaScript responsible for that, which I'll talk more about in a bit. Those wanting to peek ahead can check out the time slider source code.

Second example, a polygon editor

Our second example is a tool for editing polygons. We want users to be able to pick up a vertex of a polygon and move it somewhere else.

See those vertices? Those sure look like draggables! But these vertices are small and easy to miss, so we'd want our draggables to be bigger. We could achieve this by making the draggables invisible div elements centered on these vertices, but that gets us in trouble when the div elements are so close they overlap. At that point a click won't select the closest vertex but the top vertex.

We're going to leave those draggables in the toolbox and see how far we can get without them. As usual we start by defining a model to store the polygon we're editing.

type alias Model =
    { polygon : Polygon
    , draggedVertex : Maybe Id
    }


type alias Polygon =
    Dict Id Coords


type alias Id =
    Int

The drag starts when the user presses down on a vertex to select it. For this we'll need to calculate the vertex closest to the cursor, which we can do if we know the positions of the cursor and all vertices. Lets create a Msg type that contains those positions. We can reuse the Coords and Rect types from the previous example.

type alias Msg =
    { event : DragEvent
    , cursor : Coords
    , handlers : List ( Id, Rect )
    }


type DragEvent
    = Start
    | Move
    | Stop

Perfect! Now we can calculate the rectangle closest to the cursor when the user clicks.

closestRect : Coords -> List ( id, Rect ) -> Maybe id
closestRect cursor handlers =
    handlers
        |> List.map (Tuple.mapSecond (distance cursor << center))
        -- Find the vertex closest to the cursor.
        |> List.sortBy Tuple.second
        |> List.head
        |> Maybe.map Tuple.first


center : Rect -> Coords
center =
    Debug.todo "Implementation omitted for brevity"


distance : Coords -> Coords -> Float
distance =
    Debug.todo "Implementation omitted for brevity"

Once we have found the vertex the user picked up we have to move it to the cursor on every Move event. The resulting update function looks like this.

update : Msg -> Model -> Model
update { event, cursor, handlers } model =
    case event of
        Start ->
            { model
                | draggedVertex =
                    handlers
                        |> List.filter
                            (\( _, handler ) ->
                                distance cursor (center handler) < 25
                            )
                        |> closestRect cursor
            }

        Move ->
            case model.draggedVertex of
                Just id ->
                    { model | polygon = Dict.insert id cursor model.polygon }

                Nothing ->
                    -- The user is dragging the cursor, but nothing was picked up on the
                    -- start event. We'll sit this one out.
                    model

        Stop ->
            { model | draggedVertex = Nothing }

And that's that! Again I skipped over the JavaScript that produces our messages and I promise we'll get to that in a moment. The full polygon editor source code is available for those interested!

Last example: an outline editor

Our final example is an outline editor. An outline is a tool for organizing our thoughts on a subject, but creating a list of concepts related to the thought, each of which have their own related thoughts, and so forth. The following image shows an example outline which can be re-arranged using drag & drop. We'll keep our scope small again by not bothering with creating and deleting nodes.

We'll start by creating a model for our outline editor. It will need to keep track of two things: the outline itself and which node we're dragging.

type alias Model =
    { outline : List OutlineNode
    , draggedNode : Maybe DraggedNode
    }


type alias DraggedNode =
    -- For simplicity sake we're going to use the node's contents as an id.
    -- We get away with that here because we can ensure the nodes are unique:
    -- the user will not be able to edit them in this example.
    { node : String
    , cursorOnScreen : Coords
    , cursorOnDraggable : Coords
    }


type alias OutlineNode =
    Tree String


type Tree a
    = Tree
        { node : a
        , children : List (Tree a)
        }

Now we'll need to write behavior for the drag start, move, and end events.

The drag starts when the user pressed down on a node. We can put an onClick handler on each node to detect when this happens. We'll skip the implementation in this post, but it's part of the full outline editor source code!

Then, as the user drags a node around we need to update that node's location in the outline. This part we're going to look at in detail.

Lastly the drag stop event. We already changed the outline while the user was moving the cursor, and so all that's left to do here is change the model to its non-dragging state by setting draggedNode to Nothing.

Moving nodes in an outline

The most challenging part is to decide what the user's intention is. Are they intending to move the dragged node in front of another node, behind it, or nested beneath it?

Using dropzones we could draw invisible boxes in those positions that activate when the user moves over them, but the experience is unlikely to be great. Make the boxes too small and the user will not spend a lot of time over them, making the interface unresponsive for most of the time. Make the boxes too big and they start to overlap, causing the uppermost box to receive the dragged node even if it's not the closest. And even if we get the boxes right a future update changing page styles might move the boxes, breaking the drag & drop interaction.

Let's forget about dropzones and think about the behavior we want. There are candidate positions in the outline where we could drop a dragged node. As the user moves we'd like to display the dragged node in whichever of those positions is closest to the cursor. To figure out which position is closest we need to know where they all are. To do that we are going to put invisible elements in the DOM at each position where we can insert the dragged node. Contrary to the dropzones approach we're not going to bother giving these elements any special dimensions or positioning. We want them to just flow with the content on the page, and keep us apprised of their location. These aren't dropzones but beacons.

Apart from their coordinates in the DOM, our beacons will also need to describe their location in the outline. A beacon can define its location relative to another node in the outline.

type CandidatePosition
    = Before String
    | After String
    | PrependedIn String
    | AppendedIn String

We'll create a JSON encoder for this type so we can tag each beacon element with a data attribute containing its position in the outline. We'll then set up our JavaScript to find all elements with such a data attribute in the DOM and feed their coordinates back to us on each drag event. That will allow us to define a type for drag events containing the positions of our beacons on the screen.

type alias DragMsg =
    { cursor : Coords
    , beacons : List Beacon
    }


type alias Beacon =
    ( CandidatePosition, Rect )

Remember the closestRect function from the polygon example? It's what we need to find the CandidatePosition closest to the cursor! Once we know which candidate position is closest, all we need is a function to that moves a node to its new position in an outline. It's a tricky function to write, but it doesn't have much to do with drag & drop and so I'm skipping the implementation here. I include a solution with the outline editor source code. For those interested in some thoughts on how to approach data transformations like these, I refer to an earlier post on conversion functions, which includes an example of a similar tree manipulation problem.

Necessary JavaScript

I promised I'd get back at the JavaScript required to make these examples work. All three examples use the same JavaScript code, because it turns out they have the same needs. In every example there's one or more Html elements on the page that we need to track the position and dimensions of as a drag interaction takes place. What our JavaScript code needs to do is generate events when the mouse gets pressed, moved, and released, and bundle with those events the positions of all elements we want to track. We identify those elements by giving them a data attribute with their 'beacon ID'.

There's tons of ways to write this code and I don't believe mine is particularly insightful, so I'll not reprint it here. The draggable.js source code for these examples is available though for those interested.

Conclusion

When we need to perform a complicated task it's natural to start by looking for a library to do the heavy lifting for us. For building drag & drop interactions libraries give us draggables and dropzones but they are often a bad fit for drag & drop UIs, of which we've seen three in this post.

Are drag & drop libraries always a bad idea? I don't think so. In particular there are libraries for specific drag & drop widgets such as re-arrangable lists, for example annaghi/dnd-list. Using those when possible could save a lot of time. There are probably UIs where draggables and dropzones are precisely the right abstraction. Please send me a note if you ran into one of those, I'd love to learn about it! Drag & drop covers an incredibly broad range of functionality though, and so often an off-the-shelf solution will not be available. For those I'd put serious thought into whether draggables and dropzones are going to help build the UI or make it harder.

I recommend using 'beacon elements' if you build your own drag & drop behavior. These are regular DOM elements marked so we can access their location on every drag event. Because beacon elements don't need to do anything any element using any positioning strategy can be a beacon. This passive nature distinguishes beacons from draggables or dropzones, both of which include behavior.

I've showed a different approach to building drag & drop UIs. In this approach we subscribe to drag & drop events telling us what's happening in the DOM, update the state of our model based on those events, and finally update the screen to reflect the new state of the model. This is nothing more than the Elm architecture. I hope the examples in this post show writing drag & drop logic does not need to be an arduous task.

Thanks to Ary, Blake, Brian and Stöffel for their reviews of drafts!

A type for views

Jasper Woudenberg — Mon, 02 Jul 2018 13:00:43 +0000

This post originally appeared on the NoRedInk blog.

One of the joys of writing Elm apps is the many ways the compiler catches our mistakes, and the useful error messages it gives us when it does. It's a two-way street though, meaning that the degree to which the compiler can weed out mistakes depends on what information we disclose to it. In other words, we have to help the compiler help ourselves. In this post we'll see that this is truer for the view function than anywhere else, and we'll see how we can help the compiler help us get our view functions right.

A book recommendation application

Suppose we're building a web page showing the latest book releases of your favorite authors, ordered by popularity. The main page of the app is a simple list of books, listing the title, author name, and the ISBN of each book. Lets create a model for this application.

type alias Model =
    { authors : List Author
    }


type alias Author =
    { name : String
    , books : List Book
    , favorite : Bool
    }


type alias Book =
    { title : String
    , isbn : String
    , popularity : Int
    }

Next up is the view function of this application. For this view we need to transform the data from our model a bit. First of all we're not interested in displaying books of authors the user has not favorited, so we can filter those out. Secondly, we want to order the books of the remaining authors by popularity. Thirdly we want to display items on the page that contain data from the Book (title, isbn) and Author (name) types.

Intermezzo: Are we sure we want to do this?

But wait, maybe instead of writing a transformation function it would be easier to change our Model type to more closely resemble the structure of the page! For example, suppose our model looks like this:

type alias Model =
    { books : List Book
    }


type alias Book =
    { authorName : String
    , authorIsFavorite : Bool
    , title : String
    , isbn : String
    , popularity : Int
    }

Now our view function will be much easier to write. But this model comes with a disadvantage. Two books of the same author can disagree on whether that author is one of the users favorites. The previous model prevented this impossible state because the favorite status of an author was only stored in a single place. When faced with this trade-off we prefer our model to prevent as many impossible states as possible, even if our view function will need do some transformations.

View transformations at NoRedInk

Many of the Elm apps we have at NoRedInk have a model that looks quite different from the Html we draw on the screen. As a result our view functions will need to transform data, but that's not a problem, especially if the compiler helps make sure we're transforming correctly. But there's the rub: we'll need to work a little bit to get the compiler to help us here.

To understand why, let's look at a transformation present in every Elm app: The update function that transforms an old model state into a new one. The compiler helps us a lot with this one, it ensures that regardless of what happens our update function will always return a valid new model, which prevents all sorts of bugs. But when it comes to the view function we get no such guarantee, the compiler only ensures it returns Html. If all goes well that Html will look like our specs, but it might as well be the image of a cat as far as the compiler is concerned. Both are Html and so equally valid results of calling our view function.

To prevent such bugs we put some extra constraints on what we consider valid output from a view function. We can do this by defining a new type describing the shape of our page. Our view function we then split into two parts. First we transform our model into this View type, this is the hard part. Then we take that type and render it into Html. Because the type was designed to be similar to the structure of the page, this second part should be easy.

Let's transform some books

Let's get back to our book recommendation app and write a View type for it.

type alias View =
    List BookView


type alias BookView =
    { authorName : String
    , title : String
    , isbn : String
    , popularity : Int
    }

Now we can write the transformation function that turns our Model into a View type. This is the hardest part of the job, but because the transformation takes place between two specialized types Model and View, we will get a lot of help from the Elm compiler.

toView : Model -> View
toView model =
    model.authors
        |> List.filter .favorite
        |> List.concatMap booksForAuthor


booksForAuthor : Author -> List BookView
booksForAuthor author =
    author.books
        |> List.map
            (\{ title, isbn, popularity } ->
                { authorName = author.name
                , title = title
                , isbn = isbn
                , popularity = popularity
                }
            )
        |> List.sortBy .popularity

Finally, all that's left is to write the view function that renders our View type to Html.

view : Model -> Html msg
view model =
    toHtml (toView model)


toHtml : View -> Html msg
toHtml books =
    Html.ul [] (List.map viewBook books)


viewBook : BookView -> Html msg
viewBook book =
    Html.li []
        [ Html.text ("Author: " ++ book.authorName)
        , Html.text ("Title: " ++ book.title)
        , Html.text ("ISBN: " ++ book.isbn)
        ]

Conclusion

We've seen how we can use a View type to write our view function using some extra help from the compiler.

As a nice side-effect, this approach also makes it easy to unit-test our view logic. We can write ordinary Elm unit tests for our toView function, which contains all of our actual transformation logic. This allows us to write less tests against our Html returning view function, which although possible is harder.

Stöffel
@schtoeffel
Engineer at NoRedInk

Jasper Woudenberg
@jasperwoudnberg
Engineer at NoRedInk