DEV Community: Dmitrii Kovanikov

Pragmatic Category Theory | Part 2: Composing Semigroups

Dmitrii Kovanikov — Mon, 19 Aug 2024 07:23:43 +0000

This article was permanently moved to my personal website:

chshersh.com :: Pragmatic Category Theory | Part 2: Composing Semigroups

Pragmatic Category Theory | Part 1: Semigroup Intro

Dmitrii Kovanikov — Tue, 30 Jul 2024 18:26:33 +0000

This article was permanently moved to my personal website:

chshersh.com :: Pragmatic Category Theory | Part 1: Semigroup Intro

7 OCaml Gotchas

Dmitrii Kovanikov — Mon, 20 May 2024 08:18:24 +0000

This article was permanently moved to my personal website:

chshersh.com :: 7 OCaml Gotchas

Learn Lambda Calculus in 10 minutes with OCaml

Dmitrii Kovanikov — Mon, 05 Feb 2024 08:23:22 +0000

This article was permanently moved to my personal website:

chshersh.com :: Learn Lambda Calculus in 10 minutes with OCaml

8 months of OCaml after 8 years of Haskell in production

Dmitrii Kovanikov — Sat, 16 Dec 2023 10:15:15 +0000

This article was permanently moved to my personal website:

chshersh.com :: 8 months of OCaml after 8 years of Haskell in production

Haskell/GHC refuses to compile ugly code

Dmitrii Kovanikov — Fri, 14 Apr 2023 07:58:38 +0000

Recently, I experienced an unusual challenge when upgrading a 60K LOC Haskell project from GHC version 9.0 to 9.2, and I'd like to share my investigation journey and how I eventually fixed the problem.

Grab a cup of tea, and let's begin the journey!

TL;DR Don't write ugly code, otherwise, it won't compile later 🙅‍♀️

🐞 The Problem

The compilation of a single 200-lines testing file required too much memory, and the build step was constantly failing on CI due to the Out-Of-Memory (OOM) error.

The description of the issue is straightforward but the reasons are mysterious.

What made the situation worse, is that my local laptop has 48GB RAM while the CI runner apparently is a teapot. So I wasn't able to reproduce this problem locally.

🕵🏻‍♀️ Investigation

GHC is the Haskell compiler. It's an extremely powerful tool. It does lots of stuff, and that's why it usually requires more time and memory than you may think it should.

However, sometimes it takes unexpectedly more resources. This may happen for various reasons, including but not limited to:

A big record type (see relevant blog post for details)
A big derived instance (e.g. Generic or Read)
Lots of instances
A really big module (e.g. 20K Loc Types module)

⛰️ The gist: usually, something is really big.

In my case, nothing really seemed particularly alarming:

202-lines module with 69 lines of imports
A single testing function comprising 124 lines
No new types defined
No instances derived

You may develop an intuition for troubleshooting similar problems and be able to make pretty good educated guesses after many years with Haskell. But the most robust way to see what's actually going on is to check Core - the intermediate representation of Haskell AST after parsing, desugaring and optimising.

🔍 Core

You can view Core of a particular module by adding the following line to the top of the corresponding Haskell file:

{-# OPTIONS_GHC -ddump-simpl #-}

By default, Core is pretty ugly and unreadable. So you may need to add a few extra options to actually be able to read it.

{-# OPTIONS_GHC 
      -ddump-simpl
      -dsuppress-idinfo
      -dsuppress-coercions
      -dsuppress-type-applications
      -dsuppress-uniques
      -dsuppress-module-prefixes
#-}

Core looks like really verbose and explicit Haskell with lots of type annotations and argument passing but at least with the above options you're able to recognize some parts of it.

Here is an example from the problematic module:

-- RHS size: {terms: 46, types: 68, coercions: 18, joins: 0/0}
$wcheckUnblockedEmail
  :: LogFunc
     -> IO Connection
     -> (Connection -> IO ())
     -> Int
     -> Int
     -> Vector (LocalPool Connection)
     -> State# RealWorld
     -> (# State# RealWorld, () #)
$wcheckUnblockedEmail
  = \ (ww
         :: LogFunc
         Unf=OtherCon [])
      (ww1 :: IO Connection)
      (ww2 :: Connection -> IO ())
      (ww3 :: Int)
      (ww4 :: Int)
      (ww5 :: Vector (LocalPool Connection))
      (w :: State# RealWorld) ->
      case $wrunDBWithTransactionFunction
             (withTransaction1 `cast` <Co:14>)
             (lvl15 `cast` <Co:4>)
             ww
             ww1
             ww2
             ww3
             ww4
             ww5
             w
      of
      { (# ipv, ipv1 #) ->
      case ipv1 of {
        Nothing -> $wexpectationFailure lvl5 ipv;
        Just queuedTask ->
          case queuedTask of { QueuedTask dt ds4 dt1 ds5 ds6 ds7 ds8 ->
          case ds5 of { MagicLinkEmail ds2 ds3 ds10 ds11 ->
          case ds11 of lwild {
            __DEFAULT ->
              case dataToTag# lwild of { __DEFAULT -> lvl6 ipv lwild };
            MagicLinkEmailModeLogin -> (# ipv, () #)
          }
          }

To be able to diagnose Haskell problems with Core and understand what's actually going on, you typically spend 5 years doing GHC development or a PhD in Haskell.

In my case, I was lucky enough to come across a similar problem earlier which turns out to be a GHC bug:

https://gitlab.haskell.org/ghc/ghc/-/issues/22824

The problem was that GHC generates a lot of Core due to exponential inlining (and maybe something else). Like, really A LOT.

The Core has 3300 LOC while the original source code is 200 LOC which is 16.5x blow up (sick !!!).

As you noticed earlier, Core is really verbose. But this huge growth is too much.

💻 Code

Now, what about the code? The module has only a single test for the email auth scenario. The test is 130 lines long.

But if we look in the middle of the generated Core, we notice the following:

It's really nested
There's a pattern-matching in almost every function call and this introduces more nestedness and more matching
Records in Haskell are types with multiple fields in disguise and pattern matching on them every time results in lots of extra code
GHC also inlines library functions

And here is a part of Core in question:

After we see the problem, we need to fix it.

🔨 Fix

A quick fix would be to tune some GHC flags to prevent the compiler from doing what it's doing.

Unfortunately, after some experimentation, it turned out to be impossible.

☝️ You can't change GHC! You need to accept it for what it really is.

So I was left with only one solution: refactor code ⚒️

I suspected that GHC inlines lots of functions, so the idea behind the fix is to reduce inling. But how do I do this?

The test suite calls multiple functions in a single do-block. I can chunk this block into separate logical functions and add {-# NOINLINE #-} pragmas to those functions.

If needed, I can then move those functions to other modules to decrease the size of this particular module but this turned out to be not needed.

Luckily, after this refactoring, I noticed lots of code duplication in the module, so I was able to remove redundant code as well.

The new Haskell source code grew up to 288 LOC but the generated Core became 1900 LOC which is only 6.5x increase. Much nicer!

As an additional benefit, the relevant parts of this scenario test became more visible which makes understanding, extending and fixing the test suite in the future much easier.

Conclusion

In this post, I explained how you can use GHC Core to diagnose some non-trivial problems, and described a solution for this particular problem.

However, I'm still surprised to encounter such a problem in the first place. The offending module didn't look particular scary and didn't use any fancy GHC features. I was frustrated to see such a problem on a code many people would classify as "boring Haskell".

Moreover, the investigation of the problem required to dive into the internals of the GHC optimiser. I personally don't think that knowledge of such details for a mundane task like a GHC upgrade is a reasonable expectation from an average Haskell developer. Stuff like this doesn't help to convince people to use Haskell 😮‍💨

On the bright side, I learned something new while working on the problem. Besides, I highlighted only the problem, while I had much more enjoying moments when using Haskell!

New OSS rule: Do as little work as possible

Dmitrii Kovanikov — Sun, 26 Mar 2023 13:17:17 +0000

With time, my views on how to do OSS changed.

Previously, my motto was:

💪 Do this extra bit of work, so you won't be ashamed of the quality of your OSS projects

Now, I'm more like:

🦥 Choose the approach that requires you to do as less work as possible

I'll give an example of this.

So, in Haskell, versions of the standard library base are tightly coupled with the versions of the compiler GHC:

base-4.17.0.0 goes with GHC 9.4
base-4.18.0.0 goes with GHC 9.6
base-4.19.0.0 goes with GHC 9.8

and so on.

Previously, I always specified tight lower and upper bounds on base in my project configurations, e.g.

build-depends:
  , base > 4.15 & < 4.19

And it was done precisely due to this way of thinking: "do this extra work for extra quality".

You see, I didn't want users of my Haskell packages to compile it with the new GHC version (and a new version of base) and see errors.

They would think that my project is bad, which is not! It's Haskell who's bad for breaking my stuff!! But users don't know that. So my reputation would suffer.

Instead, I specified tight upper bounds on base, so I could say:

🙅 "You can't build my project because it doesn't explicitly support the new compiler version. I'll support it when I have time. Period."

As you can imagine, this resulted in HUGE, GIGANTIC, UNACCEPTABLE amount of work, like holy f*** 🤯🗻

For 40 packages and 3 releases of GHC, I had to open 120 PRs over time (sick !!!).

No wonder I burnt out 😮‍💨

Now, there's a solution. Versions of base in Haskell probably always will be in a form like 4.*. So I can specify the upper bound like this:

build-depends:
  , base > 4.15 & < 5

And forget about ever updating upper bounds!

In fact, this is rather common practice in the Haskell community, and lots of packages already do this stuff. But it took me a while to get here.

This approach means that if there're no breaking changes in either base or GHC, my package will continue building. And I don't have to do anything to support a new version!

If something breaks, somebody will open an issue in my GitHub repository. But I still don't have to do anything, I can just ignore it for the time being! And I can support the new version when I have the time.

So it's a win-win.

This approach is partially inspired by my recent usage of the pretty-terminal Haskell library.

It's first and only version was uploaded to Hackage in 2018 (5 years ago), and it still works without requiring any changes to it!

Now, that must feel good 😌

SQLite type system is the wooorst

Dmitrii Kovanikov — Sat, 11 Mar 2023 14:05:16 +0000

🔮 It's not a mystery that SQLite's type system is.. not its strong side. But you probably don't realise the depth of this ocean of shit 💩🌊

☕️ Grab a cup of tea and join me on this exciting journey about some "wonderful" design decisions in software engineering ⛵️

🕗 So, you want to store timestamps in your database.

Nobody has the time to read manuals. So, after browsing StackOverflow answers for about 5 minutes, you'll come up with a schema like the one below

🍶 Because software is not doomed (yet), this actually works! Good. It means we can continue with StackOverflow-Driven Development.

🔞You may actually find some time to read SQLite manuals in your life. You know, during one of those days while you're waiting for the code compilation to finish and all tweets are already written and sent.

You'll learn that SQLite doesn't actually have the TIMESTAMP type 🤯

👓 You've read the previous tweet correctly. That's right. There's no TIMESTAMP type. SQLite only can store:

NULL
INTEGER
REAL
TEXT
BLOB

You see — no TIMESTAMP 🙅‍♀️ Your entire life was a lie 🎂

🧩 And yet, things somehow work. If you like puzzles, you may guess correctly that time is stored as a value of type TEXT. And, again, this would be a lucky guess🍀

📜 A sidenote. Those strings are in the ISO8601 format. The neat thing about this format is that you can use natural text comparison aka lexicographical sorting to order your data by timestamp even though it's just text.

Aren't computers cool???

👷 If you haven't got burnout or depression after working in tech, it means that you will continue compiling code and having even more free time to read SQLite manuals.

You'll be asking uncomfortable questions like "How does SQLite decide that TIMESTAMP should be TEXT?"🤔

👴 Remember kids. Don't ask questions if you're not ready for answers. Especially in tech. 👆

Turns out, SQLite contains many surprises 💩

The resulting storage type (called affinity) is defined by checking if the type name contains one of the specified strings as substrings 🤦‍♀️

I have no words, only emotions (and they are not pleasant).

// Detect dark theme var iframe = document.getElementById('tweet-1361022421021696003-316'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1361022421021696003&theme=dark" }

The Hindley-Milner type system? Don't laugh my socks off! 🧦🤣

How about the Knuth-Morris-Pratt type system?? 😤💪

🧠💠 Once you started to piece together your brain after it had been shattered, you may even try to make sense of this world.

You may even think that TIMESTAMP must have TEXT affinity obviously!

Prepare to get f*cked so hard, you might even regret becoming a Software Engineer😨

📐Turns out, if you read the affinity rules carefully, TIMESTAMP is not TEXT. It has the NUMERIC affinity in fact!

If you're still alive after learning that a TIMESTAMP is actually NUMERIC and stores TEXT inside, congratulations 👏

But that's not the end of the story.

Listen, the thing is... A while ago, some genius decided that a column of the NUMERIC affinity can store values of any type inside.

What a piece of joke 🤡

SQLite has 5 different affinities:

🟡TEXT
🟣NUMERIC
🔴INTEGER
🟠REAL
🔵BLOB

And no, NUMERIC and INTEGER are not the same. NUMERIC is a synonym for ANY (because why not?).

Legend says, if you collect all 5 storage classes under the NUMERIC affinity in the same column, you'll be able to erase half of your data in an SQLite database with a snap of a finger 🟡🟣🔴🟠🔵🟢

🚲🏏 If you're missing good old CastFailedException, SQLite is your database of choice 👍

To summarise, any column of type TIMESTAMP, TIME, DATE, DATETIME, etc. actually has the NUMERIC affinity and can store anything inside. So you better be careful!

But let's end on a positive note🎶
People slowly but surely see the value of strong static typing even if some are still trying to slow down the progress.

Since SQLite version 3.37 or higher, you can define tables in the STRICT mode to live happily!🥳

SQLite: Strict Tables

That's all for me! Hope you enjoyed this post and had a good laugh 🤗 SQLite is actually amazing despite some design decisions!

Till the next time when I share a story with you about how I learned that it's impossible to have an in-memory DB in SQLite with multiple concurrent read-only queries 👋

This post is a copy of my Twitter thread from May 24, 2022 with a few changes

// Detect dark theme var iframe = document.getElementById('tweet-1529010669642727424-274'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1529010669642727424&theme=dark" }