The future of rbs collection

Recently I discussed the future of rbs collection in ruby-jp, the Japanese online Ruby community. The discussion was exciting, so I'd like to share it.

By the way, if you don't know rbs collection, see https://dev.to/pocke/rbs-collection-was-released-4nmm . This article introduces rbs collection with a motivating example.

TL;DR

I'm planning the following idea to scale rbs collection to grown ruby/gem_rbs_collection.

• rbs collection will migrate to use RubyGems' scoped packages in a few years, like @types/*of TypeScript.
• But RubyGems doesn't have the feature, so we need to develop it.

Background

@ybiquitous asked me did I consider providing ruby/gem_rbs_collection's RBS as a gem, like @types/** of TypeScript. I had the idea but I didn't adopt it for the first release of rbs collection. And I realized I didn't describe the reason publicly. I summarized the answer to the question in this article.

Current Implementation

Currently, rbs collection downloads RBSs with git clone from ruby/gem_rbs_collection GitHub repository.

It works well for now, but probably it will introduce many problems in a few years because the repository will grow as DefinitelyTyped.

• It will be slow!
  • As you know, cloning a large repository is slow.
  • The repository will have many gem's RBSs.
• It will be expensive for GitHub's server.
  • CocoaPods used (or still uses?) shallow clone, and it was too expensive.
    • https://github.com/CocoaPods/CocoaPods/issues/5016
    • https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935

Partial clone reduces the impact, but it just procrastinates the problems.

Why I chose git clone?

git clone is slow, but it is the simplest way.

At first I created a PoC with GitHub API v4 (GraphQL) because it could keep less API calls. However it was not a good idea. GitHub API requires an API token, so the user needs to prepare a Personal Access Token to use it for the local environment and CI.
GitHub API v4 has another problem; It truncates large file contents. Therefore I needed combining GitHub API v4 and another API.

I think git clone is a good compromise for now. It is the same phase of tsd or typings of TypeScript. We needed a simple RBS manager now as you can see from the previous article.

However git clone will introduce performance issues. I'll describe a solution idea in the next section.

Solution Idea

The solution idea is using RubyGems' scoped packages. It is the same way of @types/* of TypeScript.

For example, imagine a Gemfile:

# Gemfile

source "https://rubygems.org"

# I use the following gems
gem "rails"
gem "sidekiq"
gem "parallel"

# So, I need the following RBSs!
gem "@types/rails"
gem "@types/sidekiq"
gem "@types/parallel"

Put the Gemfile, then execute bundle install. That's all! Steep and so on load RBSs from @types/* gems.

It looks useful, but I need to solve many things to realize the idea.

First of all, we need to implement the scoped packages feature to RubyGems! Unfortunately RubyGems doesn't have the feature😭 @hsbt told a related issue to me.

Just off the top of my head I can imagine we need many tasks to implement it, adding organization, adding UI of rubygems.org, the authentication and authorization, updating gemspec/Gemfile.lock/installed package's path, and so on.

We still have another issue; It needs automatically published @types/* packages.
It is easier than implementing scoped packages, of cause. I guess we just need to set-up a workflow of GItHub Actions.

Nothing has been decided yet, but I think it is the best way. I'll investigate it.

Another idea

@znz told another idea; Using RubyGems' source instead of scoped packages. For example:

# Gemfile

source "https://rubygems.org"

# It is the same
gem "rails"
gem "sidekiq"
gem "parallel"

source "https://rbs.rubygems.org" do
  gem "rbs-rails"
  gem "rbs-sidekiq"
  gem "rbs-parallel"
end

Realizing the solution is easier than the scoped packages solution. It doesn't need to modify RubyGems, we can realize it with existing assets!

However this solution has problems:

• We need a server to host rbs.rubygems.org.
  • I don't want to maintain the server.
• The package name can conflict with a gem from rubygems.org.
  • If both rubygems.org and rbs.rubygems.org have rbs-rails gem, user cannot use both gems together.
  • We can solve the problem with more unique prefix like rbs-types-*, but I think it is not smart.

I think source is not the best way, but maybe we can use the solution as a temporary solution.

Conclusion

git clone will be slow with grown ruby/gem_rbs_collection. Therefore I'm planning to migrate rbs collection to use RubyGems' scoped package, but it has difficulty.

I think scoped package is the best solution to manage third party RBSs in the long term. I work on improving RBS management. 🚀💎

Comments

[Stan Lo]

How about using the github scope in Gemfile? I was just using something like this to test the Sentry SDK.

github "getsentry/sentry-ruby", branch: "support-exception-locals" do
  gem 'sentry-ruby', "~> 4.7.0"
  gem 'sentry-rails', "~> 4.7.0"
  gem 'sentry-sidekiq', "~> 4.7.0"
  gem 'sentry-resque', "~> 4.7.0"
end

So I imagine this would work too

github "ruby/gem_rbs_collection" do
  gem "rbs-rails"
  gem "rbs-sidekiq"
  gem "rbs-parallel"
end

It'll still use git clone AFAIK, but at least we can define type dependencies in Gemfile too.

[Masataka Pocke Kuwabara]

Ah, I didn't know the github scope😆 Thanks for sharing the idea!

I still think it should be migrated to scoped packages finally, but github scope will be a good candidate for the next step. I'll consider it too. Thanks!