DEV Community: Kirk Haines

Packing Static Files Into Crystal Binaries

Kirk Haines — Wed, 27 Oct 2021 18:10:38 +0000

Last week, I was hanging out in the Crystal Discord when what to my wondering eyes should appear? These words:

would be cool
everything in one assembly
a single executable to do everything
actually i dont think crystal can pack images and files into the result executable

With the help of Crystal macros, this very thing is quite doable. I did not realize at the time that there was an existing project that provides a version of this functionality, and so I wrote my own version.

Baked File System

I should mention the existing project. The baked_file_system shard hasn't seen maintenance since Crystal 0.35.1. Running it under Crystal 1.2.1, all of its specs pass, but it does throw an exception (which doesn't cause spec failure). So, it may or may not work out of the box right now, but if it needs fixes, they should be very modest.

The Baked File System seeks to provide a file-system-like-capability for static files that are compiled into the executable. One can treat them, more or less, like read-only files. It also automatically applies compression to the files, sacrificing some speed for space savings. It is a very neat little project, and offers a nice approach to this problem.

My Version

What I wrote is less ambitious. It doesn't seek, in any way, to emulate a filesystem approach to accessing the data. It simply packs the data into the executable, within structs that are held and organized in a hash-like way, for easy access. Files which are stored in the executable are accessed via the paths that were used when storing them, and there are methods made available to query/search the file store using just fragments of the path.

The end result is a small, fast, lean system that can be used to pack static files into an executable, and to access that data as needed. So let's take a look at just how that works.

How It Works

Crystal macros write new code at compile time. In most cases, this facility is used to generate code based on arguments passed into the macro. A standard library example of this is record macro, which is used to create a new struct, complete with predefined data fields and any methods that the programmer wants to include.

Crystal macros, however, also have several tools to let someone access other sources of data at compile time. One of these is the read_file method.##

`read_file`

This macro method reads a file and returns at StringLiteral with the contents of the file. At its core, this is all that one truly needs to inject the contents of a file into an executable:

file_contents = {{ read_file("/path/to/file") }}

Breaking it down for you, the code inside of the {{ }} is macro code, the result of which is inserted at that location, as crystal code. In this case, the macro code uses read_file to read the contents of a file, and those contents are inserted as a string, at that location.

This is very simple, and for one or even a small handful of files, there would be little reason to reach for any sort of shard to make this easier. However, as with everything, where there is a repeated pattern that involves a lot of the same code, abstractions can make that task more pleasant to work with. That is what the datapack shard is for.

`Datapack.add`

When you need to add a clearly defined set of files to the executable, you can use Datapack.add for this.

Datapack.add("/path/to/file.txt")

This macro creates a line which would be something like this:

Datapack::Data[Path.new("default://path/to/file.txt")] = Datapack::Resource.new(
        path: "/path/to/file.txt",
        data: "if this were the contents of the file, this would be the contents of the file",
        mimetype: "text/plain; charset=utf-8")

The add() macro makes a naive attempt to determine the mimetype of the file, but it only maps about 30 common mime types within the macro, so when in doubt, you can always specify the mimetype yourself:

Datapack.add("/path/to/file.txt", mimetype: "text/plain; charset=utf-8")

Also, take note of the path that is given to the Resource: default://path/to/file.txt. Every resource that is added to the datapack must have a unique path, and that path can be prefixed with a namespace. The namespace is some label, followed by a colon, and if it is not specified, it will default to default.

Namespaces can be useful to categorize assets, and are primarily implemented for programmer convenience. If they are not useful for someone's use case, they can be safely ignored, as the code will automatically ensure that a default namespace is applied in that case.

`Datapack.add_path`

In cases where there are many files to add, or where it is not known at the time that the code it written exactly which files need to be included into the compiled binary, there is another macro that one can use. This macro is Datapack.add_path, and it does just what its name suggests, adding all of the files within a given path, that match any of the given file glob patterns.

Datapack.add_path("/path/to/files", "**/*.png", "**/*.jpg", "**/*.gif")

This will add all png, jpg, and gif files within the /path/to/files directory and all of its subdirectories to the datapack.

Macro code is extremely limited with regard to what it can do. Other than the read_file method which was mentioned earlier, there are no facilities within Crystal macros to interact with the filesystem in any way. However, Crystal does provide an escape-hatch. It provides a run macro method. The method will compile and execute the file given as an argument, and whatever is returned from that execution is returned from the run call as a MacroId.

For the purpose of packing an entire directory, there is a very small utility bundled into the shard that, when compiled, yields a simple script that walks a directory, matching any of the file pattern globs that it was given (and matching */ if nothing was provided), and then returns both the path to each file, and it's MIME type, as determined by the standard Crystal MIME library. This results in a pretty robust solution for finding files to read, and for determining their MIME types. Any file with a type that still can not be determined will be labeled as application/octet-stream.

It is easy to imagine a scenario where one has a complete web site, or some application where, for business or ease-of-distribution purposes, it would be nice if the whole thing could be packaged into a single executable file, with no other dependency on any resources within the filesystem.

As an example, consider this small Kemal application:

require "kemal"
require "datapack"

module Dpex
  VERSION = "0.1.0"
end

Datapack.add_path("./assets","**/*",namespace: "assets")

get "/assets/:path" do |env|
  path = env.params.url["path"]
  resource = Datapack::Data.find?("assets:/./assets/#{path}")
  if resource
    env.response.headers["Content-Type"] = resource.mimetype
    resource.data
  else
    env.response.status = HTTP::Status::NOT_FOUND
  end
end

Kemal.run

This will compile all of the files which are in the assets/ directory into the Kemal server's executable, and any access to /assets/ will be served out of those compiled in files. All that is needed is the compiled executable file.

If you are curious, here is how the performance looks on my desktop, running Ubuntu 20.04 under WSL2.

Document Path:          /assets/bar.html
Document Length:        69 bytes

Concurrency Level:      20
Time taken for tests:   0.442 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      1690000 bytes
HTML transferred:       690000 bytes
Requests per second:    22602.60 [#/sec] (mean)
Time per request:       0.885 [ms] (mean)
Time per request:       0.044 [ms] (mean, across all concurrent requests)
Transfer rate:          3730.31 [Kbytes/sec] received

Searching the Datapack

In addition to the basic Hash-like #[] and #[]= methods for accessing the datapack, the Datapack shard offers several convenience methods for searching the files stored in the pack. It creates a simple Path based index of all of the files, based on the individual fragments of the path. There is a set of convenience methods that can search this index in order to find files with only partial path information, or to find collections of files under a given namespace or partial path.

The following methods are provided:

#[] - Returns the resource at the given path.
#find - Finds a single file by Path or a String
#find? - Finds a single file by Path or a String, returning nil if not found
#find_all - Finds all files matching a given Path or String
#find_key - Finds a single key by a Path or String
#find_key? - Finds a single key by a Path or String, returning nil if not found
#find_all_keys - Finds all keys matching a given Path or String

Usage looks like this:

file = Datapack.Data["assets:/./assets/bar.html"]
the_same_file = Datapack.find("assets:/bar.html")
asset_keys = Datapack.find_all_keys("assets:")
images = Datapack.find_all("assets:/images")

Wrapping Up

Embedding static files within an Crystal executable is a straightfoward process, using a macro and the read_file macro method. If you are embedding a lot of files, or you want convenience methods for accessing and searching the files that are embedded, a shard like [datapack.cr] or the older and more filesystem-oriented baked-file-system is a lot more convenient than writing all of the file reading boilerplate yourself.

Twitch EventSub - The Direct Approach to Getting Started With It

Kirk Haines — Wed, 02 Jun 2021 05:51:08 +0000

Start Here

You want to write something to react to Twitch Events. You look at the docs, and maybe it is a little confusing. You see a table of contents that looks like this:

Authentication

Twitch API

EventSub

PubSub

If you click around a little, you might get more confused. There are a lot of details, and there seems to be more than one way to do things, at least in some cases, and it is easy to lose track of the thread of things amidst all of the information.

You might look at PubSub, and you might think, "Wow! This seems very simple and straightforward!"

You would be right. It is. Then you might discover that it only provides access to a small percentage of the Twitch events, and things that you really want, like channel.follow notifications, are not available.

Eventually, you look more at EventSub. This is the future. This is what everyone should be using:

"Awesome!" you think. "Let's dive in and....wait, how do I use this?"

To use it, you have to subscribe to events. And to subscribe to events, you have to have authorization. And to be authorized...there are no links there, but in the left menu there is an Authentication link, so you click there, and...

OK, more steps. Registering the app. Then getting a token. Only there are 5 different kinds. Which one is needed? OK, and how do I get that? And how do I use it once I have it?

You get the picture. The details are all there, in the documentation, but it is a maze of twisty little passages, and you may spend a lot of time flipping from one page to another to piece it all together.

Fear not. I've done the flipping. I've got your back.

Step 1 -- You need to register your app

First things first -- app registration. In order for your app to interact with eventsub, Twitch wants to know about it as a unique entity. So, you need to register it.

Go to https://dev.twitch.tv/console. On the right of the page, there should be a section labeled Applications. Click on the button.

In the form that is on the next page, you have to provide a few elements of information.

Give your app a name.

Unless you know with certainty what this is or will be, just put https://localhost in, and press the Add button.

Finally, select an appropriate category for your app, and then press the Create button.

You will be taken to the Apps Console, where you will see something like this:

The next step is to click the Manage button on your newly created app. You will be taken back to a page that looks just like the one where you created the app, except that it has a few extra bits of information at the bottom:

The Client ID is public information. There is no need to hide that. The application secret, however, is, well, secret. It will be generated when you press the button. Take note of the string of characters that are revealed when you press that button, and save it somewhere else for later use. You can not see it again once you leave the page, so if you lose the secret, you will have to regenerate a new one, which expires the old one.

See? This is easy so far.

The next thing that you need to do is to generate an access token for your application, using the Client-ID and Client-Secret that you have just generated.

Step 2 -- Generate your access token

The next step is to generate your access token. There are five different types of access token, but the one that is needed for EventSub is the OAuth Client Credentials Flow token type.

This type of token is an Application Access Token, intended only for server-to-server API requests, which is exactly what is needed for EventSub activities.

To get your very own shiny, new Application Access Token, you need to make a POST request to the Twitch API. The documentation detailing what is needed is in the link above, but it has the potential to be a little bit confusing.

HTTP GET requests pass extra parameters within the query string of the URL.

GET /foo?param1=abc&param2=123

HTTP POST requests typically pass parameters within the body of the HTTP request. The examples show a POST being done with the data in the URL as a Query String.

While this is not illegal per the HTTP spec, it is not typical, nor is it required when working with the Twitch API.

If you want to generate an access key manually, you can do this using curl:

curl -d client_id=$TWITCH_CLIENT_ID \
     -d client_secret=$TWITCH_CLIENT_SECRET \
     -d grant_type=client_credentials \ 
     https://id.twitch.tv/oauth2/token

If your Client-ID and your Client-Secret are stored in a couple of environment variables, TWITCH_CLIENT_ID and TWITCH_CLIENT_SECRET, the above should work from most unix-like command lines.

What is returned is one of two things. If either your Client-ID is invalid, there will be an error like this:

{"status":400,"message":"invalid client"}

If the Client-ID is valid, but the Client-Secret is invalid, the error will look like this:

{"status":403,"message":"invalid client secret"}

If both are valid, the result will be returned in JSON something like this:

{"access_token":"q3b5n90ua7du0mgpwl149ge2yf90r0","expires_in":4776914,"token_type":"bearer"}

The value for the access_token key is your golden ticket. It is what permits you to access the rest of the EventSub API.

If you want or need your software to be able to generate an access key at will, though, you will need to issue the request and receive the response programmatically. The details of this may vary considerably depending on your programming language, but maybe I can help with a few examples:

Ruby

require "uri"
require "net/http"
require "json"

uri = URI("https://id.twitch.tv/oauth2/token")
response = Net::HTTP.post(
             uri, 
             {"client_id" => CLIENT_ID,
             "client_secret" => CLIENT_SECRET,
             "grant_type" => "client_credentials"}.to_json,
             {"Content-Type" => "application/json"})
access_code = JSON.parse(response.body)["access_token"]

Javascript

The Javascript example assumes the use of Fetch.

const Url = "https://id.twitch.tv/oauth2/token"
const Data = {
  client_id: CLIENT_ID,
  client_secret: CLIENT_SECRET,
  grant_type: "client_credentials"
}
const Params = {
  headers: { "Content-Type": "application/json" },
  body: Data,
  method: "POST"
}

let access_token = ""
fetch(Url, Params)
  .then(response => {access_token = response.json()["access_token"]}

Crystal

require "http/client"
require "json"

response = HTTP::Client.post(
  url: "https://id.twitch.tv/oauth2/token",
  headers: HTTP::Headers{ "Content-Type" => "application/json" },
  body: {
    "client_id" => CLIENT_ID,
    "client_secret" => CLIENT_SECRET,
    "grant_type" => "client_credentials"}.to_json)

access_token = JSON.parse(response.body)["access_token"].as_s

Bash

The Bash example assumes that the jq utility is installed.

DATA=`curl -d client_id=CLIENT_ID \
           -d client_secret=CLIENT_SECRET \
           -d grant_type=client_credentials \
           -s https://id.twitch.tv/oauth2/token`
ACCESS_TOKEN=echo $DATA | jq .access_token

Once you have a valid access token, the world is your oyster. The rest of the Twitch EventSub API is accessible.

A Quick Note About the EventSub API URL

The sections that follow provide examples for how to perform each of the EventSub management actions, and an astute reader will note that the URL for each of the sections is the same. All EventSub API actions operate through the same API URL:

https://api.twitch.tv/helix/eventsub/subscriptions'

What differentiates the different types of actions that can be performed with EventSub requests are the HTTP verbs that are used to perform the request and the payload that accompanies it.

Listing Subscriptions

The EventSub API provides a mechanism to see what subscriptions a client currently has and their status. This is important, because there is a limited number of subscriptions allowed per client (10000), and even failed subscription requests count against that limit. This makes it important to monitor all current subscriptions so that failed or unneeded subscriptions can be deleted.

To access a list of subscriptions, a GET request must be issues to https://api.twitch.tv/helix/eventsub/subscriptions. This request should provide Client-ID and Authorization parameters, where the value of the Authorization parameter is the access token generated earlier, with Bearer prepended to it:

Authorization: Bearer deadbeefdeadbeef

With a valid access token, the response will be a JSON payload with a data field containing a list of subscriptions, along with some fields showing the limit on the number of subscriptions, as well as how many total subscriptions the client has.

A full specification for this API request can be found at https://dev.twitch.tv/docs/api/reference#get-eventsub-subscriptions.

Creating a Subscription

This is the most complex operations when dealing with EventSub, as subscription creation also involves a verification step that allows Twitch to validate that the callback that was given in the subscription request is owned by the client that requested it, as well as a signature validation to allow the client to verify that the Twitch verification request is itself valid.

Step 1 -- Request a subscription

A subscription request is initiated by sending a POST request to https://api.twitch.tv/helix/eventsub/subscriptions with the following HTTP Headers:

Client-ID: CLIENT_ID
Authorization: Bearer ACCESS_TOKEN
Content-Type: application/json

Within the body of the request, a JSON payload with four keys, version, type, condition, and transport is expected. Each should have a value as follows:

version : Currently, this is always 1.
type : This is the name of the event to subscribe to.
condition : This is an object which itself will have a single key, broadcaster_user_id, which contains the numeric user id of the account that is requesting the subscription.
transport : This is an object with three keys, method, callback, and secret, which are used to specify the transport mechanism for Twitch to send event information. Currently only webhook is supported for the method key, though the documentation alludes to plans to support others in the future. The callback will be the URL that Twitch will contact when the subscribed-to-event occurs, and the secret should be a 10 to 100 character secret value unique to this subscription.

The secret will be used to validate the subsequent subscription request verification, so your code must remember what it sent as the secret for this subscription.

The whole package to issue a request for a channel.follow subscription will look something like this:

{
  version: "1",
  type: "channel.follow",
  "condition": {
    "broadcaster_user_id": "12826"
  },
  "transport": {
    "method": "webhook",
    "callback": "https://example.com/webhooks/callback",
    "secret": "abcdefghij0123456789"
  }  
}

Step 2 -- Receive a response indicating request status

In response to the subscription request, Twitch will send a JSON payload as a response. If the request was successfully received, the response from Twitch will be similar to that described above when listing subscriptions, except that the subscription contained in the data array will have a status of webhook_callback_verification_pending:

"status": "webhook_callback_verification_pending"

This indicates that Twitch has received and will be verifying the subscription request.

Step 3 -- Receive verification

Twitch must verify that the callback provided in the subscription request belongs to the caller. To that end, it will contact the callback URL in order to initiate a verification exchange.

It will make a POST request to the callback URL. The handler for the callback URL must be able to recognize a Twitch verification request and respond appropriately. Twitch sets a number of custom HTTP headers on the request, several of which are particularly important:

Twitch-Eventsub-Message-Id : This is a UUID representing the unique ID of this specific message. This will be used in step 4.
Twitch-Eventsub-Message-Timestamp : The timestamp is also used in step 4
Twitch-Eventsub-Message-Type : This is the message type. For a verification attempt, this will be set to webhook_callback_verification.
Twitch-Eventsub-Message-Signature : This is a HMAC-SHA256 message signature in the format of sha256=4471d611ed1f44cf2fe1d7a462fc62. This is used in step 4
Twitch-Eventsub-Subscription-Type : This is the subscription type that is being verified. From the example above, this would be channel.follow.

In the body of the request will be a JSON payload that contains another copy of the subscription, in the same format that has already been discussed, along with a challenge key, and a value in the form of a random string of letters and digits separated into clusters by dashes. The challenge will be used after the request is validated, in step 5.

Step 4 -- Validate the request's Message-Signature

The Twitch-Eventsub-Message-Signature is calculated with HMAC-SHA256 using the secret that was provided to Twitch in the original subscription request. It is a concantenation of the value of the Twitch-Eventsub-Message-Id and the Twitch-Eventsub-Message-Timestamp headers with the message body, signed using HMAC-SHA256 with the aforementioned secret.

If the calculated signature does not match the signature that was provided in the Twitch-Eventsub-Message-Signature header, return a 403 status. If it does match, continue to step 5.

Your code will probably look something like this:

calculated_signature = OpenSSL::HMAC.hexdigest(
  OpenSSL::Algorithm::SHA256,
  secret,
  request.headers["Twitch-Eventsub-Message-Id"] +
    request.headers["Twitch-Eventsub-Message-Timestamp"] +
    request.body.gets_to_end
)

signature = request.headers["Twitch-Eventsub-Message-Signature"]

if signature != calculated_signature
  response.respond_with_status(403)
else
  # Yay! The Signature was verified. Continue with processing.
end

Step 5 -- Respond to the verification request with the challenge

At this point, your code will have validated Twitch's request for verification. The only thing that is left to do is to respond to the validation request.

As mentioned previously, the JSON payload in the body of the request will have contained a key challenge. The value for this key must be returned in a status code 200 response to the Twitch request, with nothing added or changed.

A sample of how this might look is as follows:

body = request_body.gets_to_end
params = JSON.parse(body)

challenge = params["challenge"]?
if challenge
  response.status_code = 200
  response.write challenge.as_s.to_slice
  response.close
end

Step 6 -- There is no step 6

At this point the subscription is active.

Deleting a Subscription

At some point, you will want to delete a subscription, either because you no longer need the information, or because a subscription request failed, and you need to clear it out.

Deleting subscriptions is a straight forward process, fortunately.

Every subscription has a UUID that identifies it. This ID can be retrieved from the JSON response that is returned when subscriptions are listed. To delete a subscription, a DELETE request is sent to the https://api.twitch.tv/helix/eventsub/subscriptions URL, with Client-ID and Authorization headers, just as for listing subscriptions.

Handling Notifications

When an event occurs for one of the active subscriptions, Twitch will send a POST request to the callback URL with the details. The headers will be the same as was discussed above for subscription verification, except that the Twitch-Eventsub-Message-Type header will have a value of `notification.

The JSON payload for the request will contain an object with two top-level keys, subscription, and event.

The value for the subscription key will contain a copy of the subscription that generated the event.

The value for the event key will contain an object that describes the event details. The precise values that will be available in this object depend on the event type. Please refer to the Twitch documentation in order to figure out what to expect.

It is expected that the notification will be verified in exactly the same manner that the subscription request was verified, by checking the HMAC2SHA256 signature of the request before trusting it. If it is validated, it is expected that a 200 response will be sent back to Twitch to confirm that the event was received. If validation fails, it is expected that a 403 response (or other appropriate 4xx response) is sent to Twitch to indicate the validation failure.

If Twitch isn't sure that the event was received (such as a case where neither a 2xx nor a 4xx response are received in response to an event), Twitch may resend the event, so one's event handler must be able to cope if an event that was already received is received a second time.

All Of The Details Are In The API Docs

The Twitch API documentation contains all of the above details, and while they are not presented in a linear fashion that is easy to implement-your-own-code from, they are all present in full details if one hunts enough.

Please use this as a guide to get yourself going, and then refer back to the Twitch documentation for full details, as in many places some of the details have been elided in order to keep this guide as direct and simple as possible.

I stream on Twitch for The Relicans. Stop by and follow me at https://www.twitch.tv/wyhaines, and feel free to drop in any time. In addition to whatever I happen to be working on that day, I'm always happy to field questions or to talk about anything that I may have written.

A Not-So-Illustrated Guide to Translating Code From Ruby To Crystal

Kirk Haines — Tue, 20 Apr 2021 13:26:53 +0000

The Spark

Once upon a time, about three years ago, I had a great idea; I wanted to implement an auto-deployment infrastructure across a bunch of servers that I was running. When a commit was pushed into GitLab/GitHub, if the branch that received the commit was deployed on one of the servers that I was managing, the repository on that server should pull the changes, and then optionally there would be a post-update hook that could be invoked to complete the local deployment.

The deploy hooks for both GitLab and GitHub attach the repository URL to the payload, so what I decided that I needed was a simple way to drop a database onto a given system that could track the git repositories on that system. Ideally, it would be something fast, and something simple enough that it did not require any real ongoing management or resources.

Enter git-index. This is a simple Ruby app that maintains a local SQLite database of git repositories, providing a command-line interface to add, list, query, or remove entries.

One of the things that I like about Crystal is that as a compiled language, one can build single binaries for utilities without depending on having the entire installed language at one's disposal. I figured that it might be an interesting exercise to take an hour or two to translate git-index from Ruby to Crystal.

Typing Empty Arrays

Crystal has a Ruby-inspired syntax, and on a small scale, a lot of Ruby code is also valid Crystal, but there are some notable differences.

One of the first things that one will encounter when porting Ruby to Crystal is how typing changes the way empty arrays and hashes are declared. In Ruby, one does a simple assignment:

leftover_argv = []

This will result in an error in Crystal, however:

Fortunately, the exception is clear about the problem, and the solution:

leftover_argv = [] of String

Instance Variables

The next major gotcha involves instance variables. In the original code, it was written to use class methods and to declare class instance variables. In hindsight, I decided that was a poor choice. So, in this port, I decided to make everything work from an instance of GitIndex instead of operating via class methods on GitIndex itself.

The implementation of this change is basically identical to how one would do it in Ruby, so I won't go into details there, but there is an important detail to consider in regard to instance variables.

In Ruby, one can declare an instance variable at any point in the code, and it will work with no problems. Ruby is very dynamic in that regard. With Crystal, however, instance variables must be declared at the class level and must be initialized directly either at the class level or in the #initialize constructor for the class.

Failing to do so may reward one with the following type of error:

To do indirect initialization of an instance variable without earning the error message above, the instance variable has to be declared at the class level with a nilable type:

class Foo
  @bar : String?     # This is nillable.
  @qux : String = "" # This has a default but is not nilable.

In the Ruby version of the code, the following was a fine way to initialize an instance variable for the first time:

def run
  @config = Config::parse_command_line

In Crystal, that initialization must occur earlier.

There are a few different options. One is to specify the typing information of the instance variable at the class level:

struct GitIndex
  @config : Hash(String, Bool | String | Symbol) = Config.parse_command_line

That works great and has the advantage that the instance variables that exist in the object definition are all clearly visible. And because the initialization can occur at the class level, there isn't even a need to provide an #initialize method.

There is an alternative, where the initialization is done in the argument specification of the #initialize method. The actual initialization code only differs from the code above by where it is placed:

def initialize(@config : Hash(String, Bool | String | Symbol) = Config.parse_command_line)
end

This approach combines the argument specification for the constructor with assignment to the instance variable. I like this approach because it allows the code to have a default configuration handling path, but it also allows someone to circumvent that and to provide a Config instance themselves just by changing the location of the initialization.

Database Access

The only other significant stumbling block to porting this utility to Crystal was porting the code that interacts with the database.

Ruby has a variety of different gems that can be used for database access, with varying levels of abstraction from gems that offer direct low level access to a database, to gems which offer a high level ORM, like ActiveRecord or Sequel, and which typically implement their own mid level abstraction layer.

In the original code, I used the SQLite3 gem directly, which offers a simple API. After creating an instance of the class, one can use #execute to run SQL and to perform queries in general.

Crystal has a standard database interface API as part of the language specification. Different database drivers can implement their own backend capabilities, but they are all accessed through that common database interface API. Unlike the SQLite3 gem, where everything can be done through #execute, with the Crystal database API, queries are executed via #query, while other SQL is executed via #execute.

Also, because of the very dynamic nature of Ruby, the return set of a database query can be iterated over simply as a set of arrays. Because Crystal is typed, however, and Arrays have to be typed, there is a little more work that has to be done on the Crystal side. So, as an example, this code in the Ruby version:

def list_records(db)
  puts "hash,path,url"
  db.execute("SELECT hash, path, url FROM repositories") do |row|
    puts row.join(',')
  end
end

Looks like this in the Crystal version:

def list_records(db)
  puts "hash,path,url"
  db.query("SELECT hash, path, url FROM repositories") do |rs|
    rs.each do
      puts [rs.read(String), rs.read(String), rs.read(String)].join(",")
    end
  end
end

The Crystal is a little more verbose for two reasons. First, a query returns a result set that must itself be iterated over to access each of the rows in the result set, and second, each field must be read with typing information provided. Similar changes have to be made in each of the methods that queried data from the database.

Build It!

You can access the source code, or download a prebuilt Linux x86_64 binary via the GitHub project:

https://github.com/wyhaines/git-index.cr

If you have Crystal installed locally, you can build it:

Profit!

While the runtime resource usage of a CLI tool like this isn't super important, it is interesting to illustrate the difference between the Ruby version and the nearly identical Crystal version.

On a real server with 53 indexed repositories, the Ruby version's time information when executing git-index -l is:

It takes 0.15 seconds to run.

time -v can be used to capture memory usage while running, as well:

And while running, it requires 14.36M of RAM.

The Crystal version, on the same machine, with the same data:

It runs in 0.006 seconds - about 25x faster - and it also only uses 3.98M of RAM.

Crystal is a great language to port your Ruby CLI tools into!