DEV Community: Ary Borenszweig

Incremental compilation for Crystal - Part 3

Ary Borenszweig — Thu, 05 Jan 2023 13:48:40 +0000

Recap from the previous two posts

In the last two posts we saw a way to generate dependencies between files like this: if a method call is made, the file where that call happens depends on the file where the method is defined. And that dependencies are transitive because of how Crystal works.

Then we saw that the file dependencies graph is quite messy for real programs and that changing one file usually affects most of the other files.

An observation

After writing the last post I realized there's something slightly wrong in the way dependencies, and more precisely transitive dependencies, are tracked.

Consider the following files:

# foo.cr
require "bar"

def foo
  bar
end

foo

# bar.cr
require "baz"
require "qux"

def bar
  baz
  qux
end

# baz.cr

def baz
  1
end

# qux.cr

def qux
  "hello"
end

Doing a dependencies graph for the above program gives this:

So bar.cr depends on both baz.cr and qux.cr.

If we change the method qux to return an int, bar will start retuning an int, and so foo will start returning an int: transitive dependencies doing their stuff.

BUT, and this is the important bit, if we change the method baz to return a different type, no matter what type, and no matter how many changes we do to it, we do need to re-compile and re-analize bar, but it doesn't affect foo because baz is not being used as a return value of bar.

That means that the messy dependencies graph we have so far might not be that messy after all. And also, that we have two graphs: direct dependencies and transitive dependencies.

When we checked the method IO#<< and saw that it calls obj.to_s, so io.cr depends on whatever type we give to IO#<<, the method definition is this:

class IO
  def <<(obj) : self
    obj.to_s self
    self
  end
end

But note that the obj.to_s call isn't part of the return type. That means that, yes, if the file where the type of obj changes we need to recompile io.cr, but it doesn't mean we have to recompile files that depend on io.cr.

Computing actual transitive dependencies

Now, the main issue is determining which method calls affect the return type.

For example, consider this:

def foo
  x = bar
  x
end

Here the method returns x, and x came from a call to bar, so foo transitively depends on bar. So we have to keep track of how variables came to exist, and their dependencies. Luckily the Crystal compiler already keeps track of this to actually compute the type of x, so in theory this should be easy to do.

Another example is this:

def foo
  qux
  if something
    bar
  else
    baz
  end
end

Here the last expression in the method is an if, so the actual dependencies of foo are bar and baz, but not qux.

Another dependency that might not be easy to see is call arguments. For example:

def foo
  bar(baz(1))
end

Here foo depends on bar. But does it depend on baz? Well, if baz suddenly changes and returns a different type, another overload of bar might be hit that returns a different type, and so foo might start returning a different type. So it seems that call arguments need to be taken into account.

What's next?

I'll need some time to code this and get it right. Once I do that I'll post my findings. Will the dependencies graph be leaner? Will we find happiness? Will it make us cry? Stay tuned!

Incremental compilation for Crystal - Part 2

Ary Borenszweig — Wed, 04 Jan 2023 18:27:25 +0000

In the last post we found out that we could build a file dependencies graph for a given project. We also saw that, depending on the code, a file could depend on one another even if it's not immediately clear that that's the case.

But... is that dependency issue really an issue?

Checking the tool against real-world projects

In this section I'll look at the dependencies graph of two medium/big projects:

Mint
The Crystal compiler

We can use this branch of the compiler to compile Mint and the compiler to get the "dependencies.dot" files.

Then we can use this program to analyze the above files. The program will read a "dependencies.dot" file, build the graph in memory, and then for each node in that graph count how many nodes are, recursively, affected by it. This answers this question: if we make a change to a file, what are all the files that we need to re-analyze?

Transitive dependencies

Before looking at that, though, note that I said "recursively". If a file A depends on B, and B depends on C, why if we make changes to C we need to re-analyze A?

Let's take a look at an example.

# foo.cr
require "bar"

def foo
  Bar.bar
end

foo

# bar.cr
require "baz"

module Bar
  def self.bar
    Baz.baz
  end
end

# baz.cr
module Baz
  def self.baz
    1
  end
end

Here we have that foo.cr depends on bar.cr, and bar.cr depends on baz.cr.

What does foo return? It returns an integer, because foo calls Bar.bar, which in turn calls Baz.baz, which returns 1.

If we change Baz.baz to return a string, foo will start returning a string too. So, effectively, changing baz.cr affected foo.cr. And note that the program still compiles just fine.

Does this transitive dependency re-analysis happen in other languages too? I don't know. But let's imagine Crystal forces you to specify argument and return types for methods, like Java, C#, Go, Rust, and essentially every statically typed language out there.

Now we have these files:

# foo.cr
require "bar"

def foo : Int32
  Bar.bar
end

foo

# bar.cr
require "baz"

module Bar
  def self.bar : Int32
    Baz.baz
  end
end

# baz.cr
module Baz
  def self.baz : Int32
    1
  end
end

Now if we change Baz.baz to return a String, Bar will stop compiling. We can change Bar.bar to return a String and then change foo to return a String and then it will compile again. But doing that we changed all those files, so they naturally have to be re-analyzed.

But files don't always have to be re-analyzed like this. Let's say we change Baz.baz to return a String. We can make Bar still compile and return an Int32 like this:

module Bar
  def self.bar : Int32
    Baz.baz.size # The size of the string
  end
end

Here we ended up changing Bar because of changes made to Baz, but we didn't have to change foo. Not only that: we don't have to re-analyze foo because Bar.bar still returns an int. We need to re-compile bar.cr, but foo can still keep calling the same method.

With all this, another partial conclusion is that not having mandatory type signatures in methods means even more that a change in one file can affect files that are far away in the dependency graph.

Checking affected files counts

If we run the above mentioned program on the "dependencies.dot" file for Mint, we'll see something like this:

src/macros.cr: 1
src/mint.cr: 1
src/utils/index_html.ecr: 1
src/utils/service_worker.ecr: 1
src/constants.cr: 2
src/app_init/scaffold.cr: 1193
src/artifact_cleaner.cr: 1193
src/assets.cr: 1193
src/ast.cr: 1193
...
1059 lines more
...

Total files: 1405

I forgot to mention that this only outputs files in "your project". That is, the most common scenario is that you'll change files in your project, not in shards or the standard library.

Also, the output is sorted by number of affected files for a given file.

In this project there are a total of 1405 files, including files from shards and the standard library. And for the vast majority of files, if we make a change to them it ends up affecting 1193 files. That means that we'll end up needing to re-analyze most of the files anyway.

Doing this with the compiler is similar:

src/compiler/crystal.cr: 1
src/compiler/crystal/tools/doc/html/_head.html: 1
src/compiler/crystal/tools/doc/html/_list_items.html: 1
src/compiler/crystal/tools/doc/html/_method_detail.html: 1
src/compiler/crystal/tools/doc/html/_method_summary.html: 1
src/compiler/crystal/tools/doc/html/_methods_inherited.html: 1
src/compiler/crystal/tools/doc/html/_other_types.html: 1
src/compiler/crystal/tools/doc/html/_sidebar.html: 1
src/compiler/crystal/tools/doc/html/js/doc.js: 1
src/compiler/crystal/tools/doc/html/main.html: 1
src/compiler/crystal/tools/doc/html/sitemap.xml: 1
src/compiler/crystal/tools/doc/html/type.html: 1
src/compiler/crystal/tools/init/template/example.cr.ecr: 1
src/compiler/crystal/tools/init/template/example_spec.cr.ecr: 1
src/compiler/crystal/tools/init/template/gitignore.ecr: 1
src/compiler/crystal/tools/init/template/license.ecr: 1
src/compiler/crystal/tools/init/template/readme.md.ecr: 1
src/compiler/crystal/tools/init/template/shard.yml.ecr: 1
src/compiler/crystal/tools/playground/views/_workbook.html.ecr: 1
src/compiler/crystal/tools/playground/views/layout.html.ecr: 1
src/crystal/compiler_rt/divmod128.cr: 1
src/crystal/compiler_rt/fixint.cr: 1
src/crystal/compiler_rt/float.cr: 1
src/crystal/compiler_rt/mul.cr: 1
src/crystal/once.cr: 1
src/crystal/system/process.cr: 1
src/http/server/handlers/static_file_handler.html: 1
src/log.cr: 1
src/log/setup.cr: 3
src/compiler/crystal/command.cr: 6
src/compiler/crystal/command/cursor.cr: 6
src/compiler/crystal/command/eval.cr: 6
src/compiler/crystal/command/repl.cr: 6
src/compiler/crystal/command/spec.cr: 6
src/compiler/crystal/command/docs.cr: 7
src/compiler/crystal/command/env.cr: 7
src/compiler/crystal/command/format.cr: 7
src/compiler/crystal/command/playground.cr: 7
src/compiler/crystal/tools/print_hierarchy.cr: 7
src/spec/cli.cr: 7
src/array.cr: 451
src/base64.cr: 451
...
444 lines more
...

Total files: 513

Again, with a total of 513 files, making changes to most files ends up affecting 451 other files, which are almost all of them. So trying to reuse previous compilation data won't be a lot of benefit.

The end?

Is this the end of the incremental compilation exploration? Is there nothing we can do about this? No! This is just the beginning. We have to look deeper into how things depend on each other. Maybe files aren't the incremental compilation unit we are looking for? Maybe we could introduce some small changes to the language to prevent transitive dependencies? Maybe the analysis can still be reused for methods that have a specified return type?

Like before, I didn't explore any of these options yet. Maybe I will!

Incremental compilation for Crystal - Part 1

Ary Borenszweig — Tue, 03 Jan 2023 14:26:35 +0000

I strongly believe that the Crystal programming language has great potential. The only thing that, in my mind, is holding it back a bit is its slow compile times, which become more and more noticeable as your project grows.

Why is the compiler slow? And can we improve the situation? This is what I'm going to talk about here, and think about solutions as we go.

Why the compiler is slow?

The main reason is that the compiler compiles the entire program, from scratch, every time you invoke it. Well, it at least does a full semantic analysis of your program: there is some caching going on regarding producing object files. But semantic analysis is always done, from scratch, for the entire program.

And when I say "the entire program" it's the code that you wrote, but also the code from shards you use, and even the standard library. And, well, it's not the "entire program" because only those types and methods that are effectively used need to be analyzed and compiled, but it can still be quite a lot of code.

The actual algorithms used in the compiler are good and fast. We could try to make them even faster. But, as you add more and more code to your program, inevitably, no matter how fast the algorithms, it will take more and more time to compile programs.

Why is semantic analysis done from scratch every time?

Not doing semantic analysis from scratch means doing it by chunks, and reusing that previous knowledge.

Most, if not all, compiled programming languages will analyze and compile individual files and then link them afterwards. Can we do the same in Crystal?

Let's consider this file:

# dependency.cr

def add(x, y)
  x + y
end

Well, we have no idea what are the types of x and y. If we call it with two ints then we can type and compile that method with that information. And the same is true if we pass it two strings. Or any object that has a + method with an argument. So we can't type and compile that file just like that.

That's one thing. Another thing is not knowing what types resolve to. For example in this file:

# dependency.cr

def something
  Foo.bar
end

something

Here something doesn't take any arguments so we could try to type and compile it right away. But if we do so... what's Foo? And what's the class method bar in it? We can't know. Well, maybe a require is missing in that file:

# dependency.cr

require "foo"

def something
  Foo.bar
end

something

and now if foo.cr defines Foo all is good. But Crystal (and Ruby) allow not requiring a dependency like that, and instead it can come from somewhere else. For example:

# caller.cr

require "dependency"

class Foo
  def self.bar
    1
  end
end

Here caller.cr defines a class Foo with a class method bar, and requires dependency which in turns defines something that uses that Foo. And this is perfectly fine, even though there's no explicit mention of where does Foo come from in something.

The conclusion so far is that it's almost always impossible to look at a single file and type it without knowing the full program.

Why Crystal allows writing programs like that?

At this point one might wonder "is it a good idea that a file can work without it specifying where do types and methods come from?" But here I'm not going to try to answer that question, which is mainly subjective, and instead try to focus on how we could improve Crystal as it is right now. Of course Ruby could be "improved" (made faster) in many ways if we changed it (like, disallow some dynamisms, force some type declarations, etc.) but Ruby is being improved while not changing the way it is, while keeping its philosophy. Ruby has a vision, and it's being improved while keeping that vision. And I think that with Crystal we should strive to do the same.

Incremental compilation

So let's discard for now the idea of being able to compile files individually. What else can we do?

One idea is to compile the entire program first, and then try to reuse that knowledge for subsequent compilations. It would work like this:

We compile the entire program and build a dependencies graph: which files depend on which other files?
We also remember the types used in methods? For example if add(x, y) was called with two ints, we remember that it returns an int.
Next time we compile a program, if we find a call to add(x, y) with two ints, we check if the file where add was defined, or any of its dependencies (recursively) changed, and if not, we could avoid typing that method again as we would know that it returns an int.

Would that really work?

Looking at file dependencies in real programs

I created a branch that modifies the compiler to output a file dependencies graph for a given program. You can try it by checking out that branch, creating a compiler from it and then compiling any program. This will generate a dependencies.dot file which you can then turn into a PDF by using this command:

$ dot -Tpdf dependencies.dot > dependencies.pdf

(you'll need to install graphviz first)

Running it against this program:

# foo.cr
puts "Hello world!"

will produce this graph:

What a lovely tree-like graph! It's very easy to see how no cycles exist here.

They say "beauty lies in the eyes of the beholder" but I hope we can all agree that the graph above is a mess.

Still, within all that mess, we can find this:

kernel.cr defines puts, so it's natural that foo.cr dependes on it. But what's interesting is that nobody depends on foo.cr (naturally.) And that also means that if next time we only change foo.cr then we can reuse everything else from the previous compilation.

Trouble ahead

This looks promising. But running the tool with some bigger programs I found something. Let's take a look at this program:

# foo.cr

class Foo
  def to_s(io : IO)
    io << "foo"
  end
end

puts Foo.new

The graph near foo.cr now looks like this:

Now foo.cr has more dependencies, but something curious is that someone is now depending on foo.cr. Who?

We could play the "trace the line" game to find it, but looking at the dot file there's this:

  "src/io.cr" -> "foo.cr"

The above means "src/io.cr" depends on "foo.cr". WAT? How can the file that defines IO depend on "foo.cr"? It should be the other way around!

Debugging this a bit, we can see that the call the program makes is this one:

puts Foo.new

Next we have that the definition of puts is:

def puts(*objects) : Nil
  STDOUT.puts *objects
end

So that's calling puts on an IO (STDOUT). Let's take a look at that:

class IO
  def puts(obj : _) : Nil
    self << obj
    puts
  end
end

This is calling IO#<<(obj) where obj is an instance of Foo. Let's take a look at that << method:

class IO
  def <<(obj) : self
    obj.to_s self
    self
  end
end

So this is calling obj.to_s(self), where obj is an instance of Foo. And where is that method defined? In foo.cr! Bingo!

That's why suddenly io.cr depends on foo.cr.

One way to understand this is to know that in Crystal all methods are monomorphized. Monomorwhat? It just means that if you call IO#puts with a type X, a method will be generated for that particular X. If you call it with a Y, a method will be generated for that particular Y (for that particular type.) In the example above, an IO#<<(obj) method was generated where obj is of type Foo.

This is one reason that, I think, explains how messy the graph above is.

In other compiled, statically typed programming languages, this doesn't happen. Usually obj responds to some explicit interface and then the method lies behind a virtual table and a virtual dispatch. None of this happens in Crystal.

Does it matter?

Does it matter that io.cr depends on foo.cr? Maybe it's not a big deal. Well, it probably is. It means that changes to a file are likely to affect seemingly unrelated files. And so if we try to use the "reuse previous compilation" strategy, not a lot could be reused in the end.

The end?

This is not the end of the story. It's just the end of this blog post. I didn't continue thinking about how to approach this problem, but hopefully these posts spark some interest and some thoughts from others about how to solve this interesting problem.

Why I love Ruby: the secret algorithms

Ary Borenszweig — Tue, 01 Mar 2022 13:15:53 +0000

It's no secret that Crystal's standard library looks almost exactly the same as Ruby. Ruby's one has a very well designed API, with lots of useful things in it, so why not have a similar API in Crystal? Why rethink names and other things from scratch?

So a few years ago we started adding these methods as we needed them for the compiler, or just for fun and completeness. While doing that, we would always compare Ruby's performance with Crystal. The idea was that, because Crystal is statically typed and compiled, relying on LLVM optimizations, it would always beat Ruby. It turns out, that wasn't always the case! We would think "How can Ruby be faster here? What's this magic?." Well, looking at Ruby's source code we found so many beautiful and performant algorithms. And they are all hidden. Most people don't know about them.

So here I will talk about some of them.

Multiplying strings or arrays

In Ruby you can do this:

"hello" * 3 # => "hellohellohello"

What's the most obvious way to implement this? Like this:

The string consists of 5 bytes
We are multiplying it by 3, so we'll need a string of 15 bytes
Iterate 3 times by copying those 5 bytes over and over (for efficiency, we can use something like memcpy)

And that was the first implementation we did in Crystal. And Ruby was faster!

So what does Ruby do? Let's say we need to multiply the string by 16. We can do this:

The string consists of 5 bytes
We are multiplying it by 16, so we need 80 bytes total
First we copy 5 bytes from the original string. Great! 75 to go
Next we copy 5 bytes more. Great! 70 to go.
Now that we have 10 bytes ("hellohello"), we can copy those 10 bytes in the next position. Now we copied 20 bytes. 60 to go.
Now that we have 20 bytes ("hellohellohellohello") let's copy those over. Now we copied 40 bytes! 40 to go.
Next we copy the 40 bytes we have to the next position. Done! All 80 bytes copied.

This doesn't look like an optimization at all, right? After all we end up copying 80 bytes in any case. However, copying 40 bytes with a single memcpy call is more efficient than doing 8 memcpy calls copying 5 bytes. memcpy is really well optimized! I don't know what tricks it does, but it can copy large memory portions very fast.

Of course this works great when the amount with are multiplying by is a power of two. If that's not the case we can fill the remaining bytes by using the simpler algorithm.

Because I haven't read many algorithm books, maybe this is a well known algorithm? I don't know. But here's something. Ruby introduced this optimization 14 years ago, while Go introduced it 8 years ago. In both cases the first algorithm to use was the simplest one, so at least it doesn't look like this optimization is immediately obvious.

There's more!

Ruby also has another optimization, which I didn't see in Go. This isn't to prove Ruby is better than Go or anything like that, it's just to show how much care there is in every single Ruby method in the standard library, and to show that this isn't like that in every language out there.

If we are doing something like this:

"a" * 100

Ruby will notice that the string we are multiplying occupies only one byte. In that case, it does the following optimization:

Create a new string of size 100 bytes
Call memset to fill those 100 bytes with the single byte from the original string.

Arrays

Before meeting Ruby I was mainly coding in Java and C#. If you need some sort of collection in those languages you have many choices. Let's see Java:

This is also similar in Rust:

The reason there are many collection types is that each one of them has better performance in some use cases, but worse performance in others. Depending on your use case you should choose one or another.

So, before you just start collecting elements somewhere, you have to think about how that collection of elements is going to be used, by you and potentially others! Maybe you need to document "Do this, but please don't do that because it could be inefficient," and so on.

Where are all these collection types in Ruby?

One type to rule them all

In one of my many trips to Rubyland I found an old sheet that contained ancient writing. I didn't understand the meaning, so I copied the contents to a foo.rb file and ran it with ruby. This was the output:

Three Types for the Java-kings under the beans,
Seven for the Rust-lords in their halls of stone,
Nine for Mortal Men doomed to use C++,
One for the Happy Lord on his shiny throne
In the Land of Ruby where Happiness lies.
One Type to rule them all, One Type to find them,
One Type to bring them all, and in the bliss bind them,
In the Land of Ruby where Hapiness lies.

In Ruby, there's only one type for this: Array. And it's implemented in a way that covers a lot of uses cases. When you need a collection of elements you don't need to think what to use: the answer is to use Array. Simplifying users lives is what Ruby is all about.

First, it's implemented like Java's ArrayList and never like a LinkedList. A linked list looks nice in paper, but having to allocate memory for every node when you want to insert an element is very expensive! Then traversing the list is also not great, when the memory for these nodes can be scattered, so cache locally can't be used.

I don't want to get too technical, but Array is implemented by allocating some memory, let's say with an initial capacity of 10 elements, but it starts empty. Whenever you insert an element, as long as the current size didn't reach the current capacity, you do it. If not, you ask a bit more memory (let's say, 20 elements), copy what you had before into this new space, and then put the new element.

Array has methods like push and pop that let you use it like a stack, and that's easy to do with the structure described above. But then it has methods like shift and unshift that let you use it as a queue or dequeue too! Normally if you want to insert an element right at the beginning of an array you would have to move all the existing contents forward, then put an element. I'm not sure what Ruby does here, but that's definitely not what it does. It looks like Ruby knows where the array starts, so when you shift, it just moves that pointer forward. It's really efficient!

A lot more

There are a lot more operations you can do on an Array, so many that it's a lot to cover in a blogpost, and they are all handled with extreme care and thought, so you can use arrays however you need them and have a good guarantee that things will work well and fast.

Coming up next...

I don't know! I thought this post was going to be the final part, but it seems I still have many nice things to say about Ruby, so we'll see :-)

Why I love Ruby: blocks

Ary Borenszweig — Tue, 22 Feb 2022 08:55:53 +0000

If you already know what are blocks in Ruby you can skip to the "Ruby blocks are not just anonymous functions" section. Otherwise, here's a small tutorial.

Small introduction to Ruby blocks

Ruby methods can accept a block. Here's the simplest example:

def run
  yield
end

run do
  puts "Hello world!"
end

In the above example run yields. When we call run we give it a block (do ... end) Then the block's contents are executed exactly where yield was used.

With this, we can define something a bit more useful:

def twice
  yield
  yield
end

twice do
  puts "Hello world!"
end

Because we do yield twice, the block gets executed... twice!

This general idea of executing something a number of times can also be achieved like this in Ruby:

3.times do
  puts "Hello world!"
end

And blocks can receive arguments, so you can also do this:

3.times do |i|
  puts "Hello number #{i}!"
end

The above outputs this:

Hello number 0!
Hello number 1!
Hello number 2!

Ruby blocks are not just anonymous functions

At first glance Ruby blocks are similar to a concept that exists in many languages: passing anonymous functions to functions that, in turn, receive functions and execute them.

Let's try defining a times function in Go:

func times(n int, block func(int)) {
    for i := 0; i < n; i++ {
        block(i)
    }
}

This function takes an integer n and will invoke the given function named block that many times.

Here's a full program that uses it:

package main

import "fmt"

func times(n int, block func(int)) {
    for i := 0; i < n; i++ {
        block(i)
    }
}

func main() {
    times(3, func(i int) {
        fmt.Printf("Hello number %v!\n", i)
    })
}

The output is the same as the 3.times Ruby snippet mentioned before.

Aside from having to write a bit more code, and having a bit more noise in it, it might seem that blocks aren't that special in Ruby after all... you can do the above thing in Go, Java, C#, Rust, and many more.

To show that Ruby blocks are different, I came up with this program:

def greet_20_numbers
  seconds = Time.now.sec

  20.times do |i|
    puts "Hello number #{i + 1}!"
    if i == seconds
      puts "Oh no, time's up!"
      return
    end
  end
  puts "Bye!"
end

greet_20_numbers

The above program doesn't make a lot of sense, but here's how it works:

It prints 20 times "Hello number ...!" followed by priting "Bye!"
However, if while printing the above it happens that the number to greet is the number of seconds on the clock, we print "Oh no, time's up!". Note that we don't print "Bye!" in this case because we return from the method.

Let's try to translate the above program to Go:

package main

import "fmt"
import "time"

func times(n int, block func(int)) {
    for i := 0; i < n; i++ {
        block(i)
    }
}

func greet20numbers() {
    now := time.Now()
    second := now.Second()

    times(20, func(i int) {
        fmt.Printf("Hello number %v!\n", i)
        if i == second {
            fmt.Println("Oh, time's up!")
            return
        }
    })
    fmt.Printf("Bye!")
}

func main() {
    greet20numbers()
}

Running the above program doesn't work like the Ruby program. Can you spot why?

In Go we also use return to exit the function. But this return returns from the function we gave to times, it doesn't return from greet20numbers! In Ruby, return returns from the enclosing method, not the enclosing block.

This is the most important difference between Ruby blocks and passing functions around.

For example many languages (C, Java, C#, Go, Rust) have if, while or for. These constructs also have a "block" of sorts where we do things. In the case of if, whatever we put inside that if gets executed if the condition holds. For while it's similar: the "block" gets executed while the condition holds. With for you can use more complex conditions. And in all of them if you return from inside these "blocks", you return from the enclosing function. But then you can't define other constructs that behave in the same way: you might be able to pass "blocks" around, but if you use return then you return from those "blocks", not from the enclosing method.

In Ruby, with blocks, you can create your own constructs that behave like if, while or for would. If you return from inside 3.times do ... end, you return from the method. If you do:

File.open("some_file.txt") do |file|
  file.each_line do |line|
    return if some_condition
  end
end

then that return which is nested inside two blocks will return from the enclosing method! Then all of these blocks don't look like anonymous functions anymore, they look like augmented language syntax.

Give me a break

There's one more thing you can do inside blocks that you can't do with regular anonymous functions from other languages: you can break from them.

For example:

3.times do |i|
  puts "Hello number #{i}!"

  break if i == 1
end

The above will print "Hello number 0!", then "Hello number 1!" and stop.

Just like you can break from a while in many languages, breaking from loops lets you create your own loops that enjoy all the benefits of regular while loops: you can return from them and you can break from them! This further reinforces the notion that these methods look like new language constructs rather than anonymous functions.

Finally, you can call next inside a block to go to the next iteration, just like you can call next or continue inside a while in other languages. However, you can do this with anonymous functions: just return from them.

There's more to Ruby blocks

Ruby blocks can also be captured and passed around. But I won't cover that here because that does look like anonymous functions or passing functions around. That said, you can also execute blocks in the context of another object, which again I won't cover here. But all of these things make Ruby blocks even more versatile, useful and fun to use!

Other languages that have blocks

Of course Crystal has blocks and they can work like Ruby. Here's the sample program in Crystal:

def greet_20_numbers
  seconds = Time.local.second
  puts seconds

  20.times do |i|
    puts "Hello number #{i + 1}!"
    if i == seconds
      puts "Oh, it's time to get back to work!"
      return
    end
  end
  puts "Bye!"
end

greet_20_numbers

The only difference is that we use Time.local.second instead of Time.now.sec, but otherwise it works exactly the same as Ruby.

Kotlin is another language where you can also augment the syntax with methods that receive functions or "blocks" and where using return actually returns from the enclosing function. This is great! I think Kotlin is a really good language that, and this is just a guess, took a lot of inspiration from Ruby. That said, it seems you can't break from these like you can in Ruby but there are small workarounds you can use for that.

Coming up next

I'll be talking about Ruby's secret algorithms.

Why I love Ruby: a great standard library

Ary Borenszweig — Mon, 21 Feb 2022 11:37:12 +0000

In many programs it's very common to want to read a file and process the contents line by line. Here's what you needed to do in Java some time ago (around the time I found out Ruby) to do that:

BufferedReader reader;
try {
  reader = new BufferedReader(new FileReader("/path/to/file.txt"));
  String line;
  while ((line = reader.readLine()) != null) {
    System.out.println(line);
    // read next line
    line = reader.readLine();
  }
} catch (IOException e) {
  e.printStackTrace();
} finally {
  reader.close()
}

It works, but there's a lot to read and write, and the intent of the code is not immediately clear.

Here's how you do it in Go, based on this StackOverflow answer:

file, err := os.Open("/path/to/file.txt")
if err != nil {
    log.Fatal(err)
}
defer file.Close()

scanner := bufio.NewScanner(file)
// optionally, resize scanner's capacity for lines over 64K, see next example
for scanner.Scan() {
    fmt.Println(scanner.Text())
}

if err := scanner.Err(); err != nil {
    log.Fatal(err)
}

Again, it works but it involves a few steps.

Here's how you do it in Rust, based on this StackOverflow answer:

let file = File::open("foo.txt")?;
let reader = BufReader::new(file);

for line in reader.lines() {
    println!("{}", line?);
}

It's actually pretty concise, but it still involves a BufReader.

Here's how you do it in Ruby:

File.foreach("/path/to/file.txt") do |line|
  puts line
end

So simple and concise! You want to read a file line by line, so you just tell Ruby to do that. No intermediate steps. No intermediate concepts like buffered readers.

Now, I know what you are thinking: in Java, Go, Rust, and probably any language, you can define a helper function to do that. Then the code ends up being as concise as Ruby.

But here's the thing: Ruby had this in the standard library right from the get-go. There's no need for all of us to write the same helper function in our projects.

And this was just about reading files line by line. When you start to use Ruby, you realize how many useful things are there in the standard library.

A comment about Java

Recent Java versions added helper functions to the standard library to read files line by line. Now it seems you can do this:

try (Stream<String> lines = Files.lines(file, Charset.defaultCharset())) {
  lines.forEachOrdered(System.out::println);
}

Neat! I think this shows that eventually they realized that this was a common operation, and that it made sense to have this somewhere in the standard library. Ruby was a pioneer.

The hidden work Ruby does for us

So to read files line by line in Java, Go and Rust we needed to open a file and then use something else on top of that: a buffered reader in Java and Rust, and a bufio.Scanner in Go. What's going on?

Well, the operating system interface for reading things from a file is fill out this chunk of bytes with what you've got, and tell me how many bytes you put there. You can see this interface bubbling up into existing programming languages:

In Java it's the read method of InputStream
In Go it's the Reader interface
In Crystal it's the IO#read method

Then reading a file line by line has to be built on top of this interface.

However, doing these primitive calls involve system calls, and doing many system calls is very expensive! So one way to deal with this is to introduce an intermediary, a buffer: the buffer will read as much as possible from the file, and then we can process the buffer's contents in-memory. In this way we produce the least amount of system calls.

When you open a file in Go or Java, they are just handles to files. If you want to use them efficiently you need to put a buffer on top of them.

What did Ruby do? Remove the need to know this, and all this complexity, from users. When you open a file in Ruby, it internally has a buffer. It's already doing what you are most likely to be doing in other languages in a more complex way! Thank you, Ruby!

Why did Ruby do this? I don't know. One guess could be reducing complexity from users, which it tries to do all the time. Another guess could be: if to use files efficiently you'll have to use a buffer in 99% of the cases, why not make them buffered by default?

IO objects in Ruby are even more powerful!

But wait, there's more!

Let's say you need to read each line of a file... but this file is encoded in a non-UTF-8 encoding, for example EUC-JP.

It's very likely that in most languages you will need to introduce a new intermediary object to do the encoding, so you'll need a few more lines of code for this. It's probably fine.

In Ruby, you can directly set in which encoding you want to read or write things, like this:

File.foreach("/path/to/file.txt", encoding: "EUC-JP") do |line|
  puts line
end

That's it!

The rest of the standard library...

This post was just about one thing: reading files line by line. But Ruby's standard library has things like these in many places. In the end you can get a lot out of the standard library by writing very little code, and also by not needing to use external dependencies.

The community

It's my impression that the Ruby community tried to do Ruby gems by mimicking the standard library way: simplify a user's life, provide helpful methods to do things by needing as few lines as possible.

Coming up next...

I'll be talking about a feature that's almost unique to Ruby: blocks!

Why I love Ruby: powerful calls

Ary Borenszweig — Sun, 20 Feb 2022 17:59:35 +0000

Designing the perfect API

In the last part I talked about how in Ruby method calls are used a lot and in many different ways. If users are going to write calls all the time, well, they better be nice to work with!

It turns out, Ruby calls are extremely flexible and powerful. Let's take a look at a few examples:

# No arguments, splits by whitespace
"a  b  c".split # => ["a", "b", "c"]

# With a string
"a,b,c".split(",") # => ["a", "b", "c"]

# With a string and a limit
"a,b,c".split(",", 2) # => ["a", "b,c"]

# With a block
"a,b,c".split(",") do |piece|
  puts piece # prints "a", "b" and "c" in order
end

# With a block and a limit
"a,b,c".split(",", 2) do |piece|
  puts piece # prints "a", then "b,c"
end

# With a regular expression
"a,,,b,,c".split(/,+/) # => ["a", "b", "c"]

# With a regular expression and a limit
"a,,,b,,c".split(/,+/, 2) # => ["a", "b,,c"]

"a\nb\n".each_line do |line|
  p line # prints "a\n", then "b\n"
end
"a\nb\n".each_line(chomp: true) do |line|
  p line # prints "a", then "b"
end

File.join("usr", "mail")          # => "usr/mail"
File.join("usr", "mail", "gumby") # => "usr/mail/gumby"

These are just three methods, namely String#split, String#each_line and File.join, but we can already see a lot of what methods support:

They can behave differently depending on the number of arguments given. In this case, calling split without any arguments splits by whitespace. Then one can pass an extra integer to specify the maximum number of pieces to return.
They can behave differently depending on the type of argument. We can give split a String, but also a Regexp.
They can also behave differently if a block is given. In the case of split, if a block is given then each piece is yielded to the block.
Some arguments can be specified with a name, like in the case of chomp, which drops a trailing newline from a string, if there's any.
Some methods can accept a variable number of arguments: File.join will join all arguments given, no need to put them first in an array

From a user's perspective, if they want to split a string, all they need to do is call split. Maybe they want to use a string for that. Maybe a regular expression. Maybe with an optional limit. Or maybe they don't need to capture things in an array and it's fine if each piece is yielded to a block. It's fine! In all these cases the operation is a split.

I really like this, because:

The operation is really a split. Why have possibly multiple names for the same operation?
Given how flexible this is, it means that if you'd like to design an API, you can choose exactly how users are going to call it. If you prefer arguments to be named, you can do it. If you want optional arguments, you can do that too. You can design the API you had in mind! The language rarely constrains you here.

What about other languages?

So now I'm going to compare this with other languages. If I like this in Ruby it must mean that not every language works like that. I'm just noting that in this series of blog posts I compare Ruby with others just to say "I really like how Ruby handled this" and not "I really dislike how this other language did it."

Java has method overloads so you can define methods that operate on different number of arguments and different types. Then Java doesn't have default arguments, but you can kind of work around it by defining two overloads and having one method call the other one with a default value. Then Kotlin, which compiles to Java bytecode (among other things) supports default arguments and named arguments. Nice!

C# started being very similar to Java but slowly evolved in a great direction: they now support default argument values and even named arguments. Pretty cool!

Go has no overloads, no default argument values and no named arguments. If we want to split a string we have:

Split: splits by a string
SplitN: splits by a string returning at most N results.

Then if they wanted these to work with a regex they would need to add SplitRegex and SplitRegexN. If they want to add more options to split they will have to combine all these names and options into the function name. I know, I know, it's not a big deal. But I like it better in Ruby. In Ruby if I need to add a limit, I just add it. In Go I need to add it too, but then I need to go back and change the function name to add N. And of course I might forget what all these functions are named... in Ruby it's always split!

Rust is also a recent, modern language that has no default arguments and no named arguments.

Then we have Haskell and Elm, which are a completely different story. In Haskell and Elm you can call a function with less arguments than those required, and that's fine. Yes, you read that right. What happens is that the result is another function that expects the missing arguments. I think this is called partial application. Partial application is very powerful and useful! But because it's how the language works by default, this happens:

If you forget to pass an argument you don't get an error message at the call site. You will get it later when you try to use that result value, where you wanted something but you got a function. This makes understanding what went wrong really hard.
You can't have default arguments: if you don't pass an argument, should the default value be used, or is it partial application?
You can't have overloads based on the number of arguments for a similar reason than the last point

In practice having less flexibility in these cases also means having to think about how to name your functions. I know of Haskell functions where ' or '' appended to the name to support different number of arguments or different types. This in my opinion leads to code that's harder to understand or easily memorize.

In languages without default arguments people would come up with ways to simulate them, which means there's some boilerplate and also a chance that everyone has their own way of doing it, leading to less code uniformity.

One more thing about Ruby methods...

There's more! In Ruby method names don't have restrictions. For example, you can't have a function called void in Java, it's a keyword. You can't have a function called type in Go: it's a keyword! I recently found out that in Elm you can't have a record field be named type, even though, unless I'm mistaken, there would be a way to allow that without a problem.

So this is Ruby:

class Foo
  def def
    1
  end
end

Foo.new.def # => 1

You can use names like begin, end, class... it's all good! Calling these requires a dot, so there's no ambiguity. Actually... you can also call methods that don't have a receiver (something before the dot,) but then you can do self.def in those cases and it will work.

Crystal

Because we really like how flexibly Ruby is, we wanted the same thing in Crystal. And so, Crystal supports default arguments, named arguments, variable number of arguments, overloading based on different types, and overloading based on whether a method receives a block or not. You can even define methods that require arguments to be passed by name, and overload based on different names:

# Arguments that come after a `*` must be passed by name
def foo(*, x)
  "I got an x: #{x}"
end

def foo(*, y)
  "I got a y: #{y}"
end

foo x: 1 # => "I got an x: 1"
foo y: 2 # => "I got a y: 2"

In fact, the very first code snippet in this post compiles and runs fine in Crystal, and it behaves almost exactly like in Ruby (in Crystal the chomp argument is true by default, while in Ruby it's false.)

In practice, you get all the flexibility that Ruby has, which lets you define delightful APIs.

Coming up next

It's time we talk about how great Ruby's standard library is!

Why I love Ruby: almost everything is a call

Ary Borenszweig — Sat, 19 Feb 2022 09:57:46 +0000

Right on the main page of the Ruby language there's this beautiful quote from Matz:

Ruby is simple in appearance, but is very complex inside, just like our human body.

It's Ruby's simplicity that I really like.

If you ask me "What do you see when you see Ruby code?" my answer is method calls.

Take a look at this code:

require "set"

class MySet
  include Enumerable

  attr_reader :inner_set

  # ...
end

There are some things here:

we require some external code
we declare a class with the class keyword
we include functionality from another module with include
we declare a read-only property with attr_reader

It turns out, only class is special here: it's a language construct that lets you define classes. But require is a method call defined by Ruby, and the same is true about include and attr_reader. They look like keywords, like something special, but they are just regular methods.

This ability of having almost everything look the same, in a consistent way, is what lets you, the Ruby user, also define things that look like language keywords, but are just ordinary methods.

In fact, it's very easy to define attr_reader ourselves. Because attr_reader already exists, lets use the name getter for that:

class Module
  def getter(name)
    symbol = "@#{name}".to_sym
    define_method(name) do
      instance_variable_get(symbol)
    end
  end
end

class Point
  getter :x
  getter :y

  def initialize(x, y)
    @x = x
    @y = y
  end
end

point = Point.new 1, 2
point.x # => 1
point.y # => 2

And there we have it!

Well, using getter is simple. Maybe defining it is a bit more complex... but it's definitely possible!

Maybe the additional feature of not having to put parentheses around method calls makes this look more like a keyword than a method. Imagine if parentheses were required for method calls:

require("set")

class MySet
  include(Enumerable)

  attr_reader(:inner_set)

  # ...
end

They don't look like keywords anymore, right? My guess is that allowing to omit parentheses was done for exactly this reason, but of course you can omit parentheses anywhere because Ruby doesn't know if you are going to use something "like a keyword" or not.

This of course is used in Ruby a lot. Here's an example of Rails' ActiveRecord:

class User < ApplicationRecord
  has_many :posts
end

To be honest, a snippet like that might have been one of the first code snippets I stumbled upon when I was learning Ruby together with Ruby on Rails. With this, you can kind of create your own mini-language for users. And all these languages will be relatively easy to use: they will all be based around method calls. Of course the names will vary, or the number of arguments. But in the end they are all method calls.

One other language that did this, and it actually did it better, or more consistently, is Elixir. In Elixir you define a module like this:

defmodule SomeModule do
  # ...
end

You define a function like this:

def method do
end

You use an if like this:

if something do
end

Note that there's always a do in the end. These are all functions defined by Elixir! (well, technically they are macros) For example, here are the docs of defmodule, from which you can jump to the source code. This is brilliant!

Like in Ruby, this also means that in Elixir you will be using the same small language (calls) for almost everything.

In other languages you might be able to do the same thing, but there might be other syntax involved. For example in Rust you call macros with a bang at the end:

format!("Hello, {}!", "world");

In Julia you use @ to call macros:

macro sayhello()
  return :( println("Hello, world!") )
end

@sayhello()

In D you can use the mixin keyword to generate code at compile-time:

template GenStruct(string Name, string M1)
{
    const char[] GenStruct = "struct " ~ Name ~ "{ int " ~ M1 ~ "; }";
}

mixin(GenStruct!("Foo", "bar"));

This might seem like a little thing, but from a user perspective you have to know what you are using. "Oh, I'm using a macro so I have to do it this way." In Ruby and Elixir it's just "I call it."

Carole King couldn't have said it better:

Winter, spring, summer or fall
All you have to do is call

Because with Ruby... you've got a friend 😌

For Crystal we considered having a different syntax for invoking macros, but in the end we used the same syntax as for method calls. And we really like the end result! Take a look at this Crystal code:

record Point, x : Int32, y : Int32

point = Point.new(1, 2)

It looks like record is a keyword that defines a type with two properties. But it's actually a macro, here's the documentation.

As a comparison, Java 14 added records to the language (where did they get that name from?! 😮) by introducing a new keyword:

record Rectangle(float length, float width) { }

If Java had a way to reduce such boilerplate right in the language itself, it wouldn't need to introduce keywords: they could just do it with the language itself. As a bonus things would get automatically documented in the API docs.

One other benefit of having such constructs be methods or macros in a language is that, at least in Ruby and Crystal, a user can redefine them. For example zeitwerk redefines require and makes it work in a different, better way. In what other language can you do this?

Coming up next...

In the next blog post I'll be talking about being able to design that perfect API.

Why I love Ruby: string representation

Ary Borenszweig — Fri, 18 Feb 2022 12:39:31 +0000

String representation

In Ruby every object responds to to_s with a good default. There's also inspect which usually reveals the internal structure of an object.

For example:

class Point
  def initialize(x, y)
    @x = x
    @y = y
  end
end

point = Point.new(1, 2)
point.to_s # => #<Point:0x00007fab148d55d8>
point.inspect # => #<Point:0x00007fab148d55d8 @x=1, @y=2>

Okay, maybe to_s isn't that useful and inspect is much more useful, but it's nice that they are there.

The nice thing about inspect is that they also takes potential cycles into account:

class Person
  def initialize(name)
    @name = name
    @sibilings = []
  end

  def add_sibiling(person)
    @sibilings << person
  end
end

ary = Person.new("Ary")
gabriel = Person.new("Gabriel")
ary.add_sibiling(gabriel)
gabriel.add_sibiling(ary)

ary.inspect # => #<Person:0x00007fe670851160 @name="Ary", @sibilings=[#<Person:0x00007fe6708510c0 @name="Gabriel", @sibilings=[#<Person:0x00007fe670851160 ...>]>]>
gabriel # => #<Person:0x00007fe6708510c0 @name="Gabriel", @sibilings=[#<Person:0x00007fe670851160 @name="Ary", @sibilings=[#<Person:0x00007fe6708510c0 ...>]>]>

Not only it doesn't crash: it also shows the object ID of objects so you can know that when there's a cycle, what that object is.

Similar code works fine in Crystal too:

class Person
  def initialize(@name : String)
    @sibilings = [] of Person
  end

  def add_sibiling(person)
    @sibilings << person
  end
end

ary = Person.new("Ary")
gabriel = Person.new("Gabriel")
ary.add_sibiling(gabriel)
gabriel.add_sibiling(ary)

ary.inspect # => #<Person:0x107d3aea0 @name="Ary", @sibilings=[#<Person:0x107d3ae60 @name="Gabriel", @sibilings=[#<Person:0x107d3aea0 ...>]>]>
gabriel.inspect # => #<Person:0x107d3ae60 @name="Gabriel", @sibilings=[#<Person:0x107d3aea0 @name="Ary", @sibilings=[#<Person:0x107d3ae60 ...>]>]>

The above output is a bit hard to read... so Ruby and Crystal allow you to pretty print objects, every type of object, right out of the box. Here's Ruby with the above value:

require "pp"
pp ary

Output:

#<Person:0x00007ffad4076fd0
 @name="Ary",
 @sibilings=
  [#<Person:0x00007ffad404b8a8
    @name="Gabriel",
    @sibilings=[#<Person:0x00007ffad4076fd0 ...>]>]>

Here's Crystal:

pp ary

Output:

#<Person:0x101418ea0
 @name="Ary",
 @sibilings=
  [#<Person:0x101418e60
    @name="Gabriel",
    @sibilings=[#<Person:0x101418ea0 ...>]>]>

In other languages this ability to easily inspect objects out of the box is not present, or is not great. At least in some languages all objects can be converted to a string, even if the output isn't very useful. But some languages doesn't do that... and I'll mention Haskell now, maybe because I'm using it at work, but I think this is also true in Rust.

In Haskell you can't turn any value into a string by default. The type has to implement the Show typeclass. In practice this means that if we have the Point type I mentioned above:

❯ ghci
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
Prelude> data Point = Point { x :: Int, y :: Int }
Prelude> p1 = Point { x = 1, y = 2 }
Prelude> p1

<interactive>:8:1: error:
    • No instance for (Show Point) arising from a use of ‘print’
    • In a stmt of an interactive GHCi command: print it

You can't see what's p1. You again have to put deriving (Show) at the end of the declaration, or define a custom show function for the Show typeclass.

This might not sound like a lot to do, but when you are debugging code and you can't inspect objects and then you have to stop what you are doing and what you were thinking about, to go and open a file and add deriving (Show) in a lot of places just to see what's going on, it's not fun. It's less fun when those types aren't in your control and it's harder to add a Show for them. At the end of this process you end up thinking "should I leave these deriving (Show) or should I remove them now that I'm done with them", and it's where my question "why aren't these derived by default" comes to mind.

This is also when Ruby's goal, "developer happiness", comes to my mind. Doing all of the above isn't fun. In Ruby we don't have to do that, Ruby took care of that and we can just have fun solving more interesting problems.

That said, once you add deriving (Show), Haskell works fine, and in general show is very well implemented for "standard library" types:

❯ ghci
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
Prelude> data Point = Point { x :: Int, y :: Int } deriving (Show)
Prelude> p1 = Point { x = 1, y = 2 }
Prelude> p1
Point {x = 1, y = 2}
Prelude> [p1]
[Point {x = 1, y = 2}]
Prelude> ["hello", "world"]
["hello","world"]

Another language as an example: Go

Let's take a look at another language. I randomly chose Go because it's a relatively modern language, very popular, and there's a playground to try things out.

Here's an example from Go's tour about arrays:

package main

import "fmt"

func main() {
    var a [2]string
    a[0] = "Hello"
    a[1] = "World"
    fmt.Println(a[0], a[1])
    fmt.Println(a)

    primes := [6]int{2, 3, 5, 7, 11, 13}
    fmt.Println(primes)
}

This is the output:

Hello World
[Hello World]
[2 3 5 7 11 13]

Let's compare it to the output of Ruby or Crystal. I'm using p here which invokes inspect on objects.

a = ["Hello", "World"]
p a[0], a[1]
p a

primes = [2, 3, 5, 7, 11, 13]
p primes

Here's the output:

"Hello"
"World"
["Hello", "World"]
[2, 3, 5, 7, 11, 13]

The first thing to note is that Ruby and Crystal put quotes around strings (well, this isn't the case if you call to_s on a string, only inspect.) Then note that array contents use inspect. This makes it possible to know that the first array has two strings. In Go it's not clear: are there two elements, "Hello" and "World", or is it just one string "Hello World"?

Another nice thing is that you can copy that array output from Ruby and Crystal, paste it into a program and it will work. This isn't generally true, but it works really well for primitive types, arrays and hashes, which are used a lot! In Go this isn't true. There aren't even commas! I don't know why.

Now, I'm sure there's a way to show arrays or strings in Go in a better way. For example in Java you can use Arrays.toString. But not having that as a default adds friction. It's not the most intuitive thing you would expect to happen.

Another thing is consistency and uniformity. In Go there's the Stringer interface that defines the String() function to turn objects to a string. So what happens if we call String() on an array?

package main

import "fmt"

func main() {
    a := [2]string{"Hello", "World"}
    fmt.Println(a.String())
}

We get a compile error:

./prog.go:7:15: a.String undefined (type [2]string has no field or method String)

So it seems only fmt knows how to turn arrays into strings, and we have to rely on this package for that, but for other types we probably should use String().

It's those things that rarely exist in Ruby that I really appreciate. When something works in one way in Ruby you understand it and think "well, I guess this also works for these other types, or in these contexts" and that's almost always true. When that's not the case in a language, it's the moment you start building a collection of exceptions in your head, and when you start looking for answers in StackOverflow.

Coming up next...

I'll talk about making a language your own.

Why I love Ruby: equality

Ary Borenszweig — Thu, 17 Feb 2022 13:42:18 +0000

In this series of posts I will list the things that I always loved about Ruby. Some of these work differently in other languages or they don't exist at all. I will try to give some comparisons when possible. But in general my feeling is that Ruby always does the most intuitive thing a programmer would expect, and I love that! And it's what I tried to preserve in Crystal.

Comparing objects

In Ruby you can compare any two objects and Ruby will do the most intuitive thing.

1 == 1           # => true
1 == 1.0         # => true (in math too!)
[1, 2] == [1, 2] # => true

The last one about comparing arrays is important: not every language does this. For instance, if you compare two arrays in Java or C#, you only get true if they are the same array, that is, the same reference to memory. Then if you search "how to compare two arrays in Java/C#" you will get lots of answers in StackOverflow trying to answer this. In Java you can use Arrays.equals, but that doesn't do a deep comparison, so you also have Arrays.deepEquals.

The tricky part about array comparison is that you might end up having an array with an object that in turn has a reference to the same array, leading to a cycle. When trying to compare them, if you don't take cycles into account, the program will stack overflow, or crash. You don't want that in your program!

Well, in Ruby it just works. It will compare two arrays by calling == on each element, and it will also take care of cycles. The following works:

a = []
a << a # => => [[...]]

a == a # => true

All of the above work just fine in Crystal, except that for the recursive array definition. I thought it worked fine in Crystal but it seems it also leads to a stack overflow. But it's something we'll eventually fix.

In Ruby you can also compare Hash, Set, and many other types, where the definition of equality is the obvious one, and the same is true in Crystal.

Equality everywhere

Another nice thing about equality in Ruby is that you get sensible defaults, and you can compare any two objects out of the box. For example, Ruby has a Struct type that's used to pack some values together, kind of like a quick definition of an object with some properties:

Point = Struct.new(:x, :y)

p1 = Point.new(1, 2)
p2 = Point.new(1, 2)
p1 == p2 # => true

In Crystal you can also define structs, and the default comparison just delegates to comparing the fields:

struct Point
  def initialize(@x : Int32, @y : Int32)
  end
end

p1 = Point.new(1, 2)
p2 = Point.new(1, 2)
p1 == p2 # => true

This all sounds very intuitive. Why would a language not do that? Well, I don't know. Take for example Haskell, where these types of structs are really common:

❯ ghci
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
Prelude> data Point = Point { x :: Int, y :: Int }
Prelude> p1 = Point { x = 1, y = 2 }
Prelude> p2 = Point { x = 1, y = 2 }
Prelude> p1 == p2

<interactive>:4:1: error:
    • No instance for (Eq Point) arising from a use of ‘==’
    • In the expression: p1 == p2
      In an equation for ‘it’: it = p1 == p2

Haskell gives a compile-time error because == is not defined for Point. We need to define one. And we can use a magic deriving to let the compiler do the obvious thing:

❯ ghci
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
Prelude> data Point = Point { x :: Int, y :: Int } deriving (Eq)
Prelude> p1 = Point { x = 1, y = 2 }
Prelude> p2 = Point { x = 1, y = 2 }
Prelude> p1 == p2
True

Now, this last point is controversial (maybe this entire blog post is controversial!) Someone might say:

But what if you didn't want this comparison by default? It's dangerous! Then you would have a bug in your code. It's better to always think about these cases and make sure the equality you want is really that one.

That's fine. My answer to that is that you want to have tests to verify that you code works in the way you want it. Compile-time safety is good, but it doesn't prove behavior.

Coming up next...

In the next part I'll talk about Ruby's to_s method.