DEV Community: Nikola Buhinicek

Engineering Ladder - Being in The Middle Sucks?

Nikola Buhinicek — Tue, 23 Jan 2024 07:59:12 +0000

You might be wondering to what "middle" am I referring to?

It's the middle I'm currently in, the middle of the Engineering Ladder, focusing on the Tech Lead role. This role sits between the Developer/Individual Contributor and "higher" roles like Staff Engineer (SE) or Engineering Manager (EM), combining aspects of both.

A Tech Lead is still contributing to the code base and has its own fair share of tasks/milestones/deliverables, but he is not anymore working "on its own". He has a team of people to lead, to watch over the codebase/architecture, to approve technical specifications, to watch over the product and more.

As I'm diving into this role, I wanted to give you my cent or two on what I think is the most interesting part of the role "upgrade". It's about the Tech Leads relationships to both the Developers on one and the SEs and EMs on the other side.

Just to be clear, this won't be 1:1 to all Tech Lead roles, their responsibilities and their descriptions. This is just my point of view on being a Tech Lead at Productive and the way we see this role.

Tech Lead - baby EM?

If you read books or posts about management, you probably read something like:

Being a Tech Lead is an exercise in leading without authority.

The without authority part is pretty true imo. Your work starts to depend on other people but at the same time you don't really have some authority over them. You can and must have the situation in control but if the situation gets out of hand, you can't resolve it on your own - you need your EM. Again, this depends on the level that your company expects you to work on as a TL.

For me and the way I see it, the most interesting and challenging part of being a Tech Lead is exactly that, the subtle intro to a management role.

It's a role where you should really start thinking a lot about the people you work with.

Several relations come into play and thats mostly bcz you have to watch both up and down from where you're standing.

Looking up the 🪜 - Staff Engineers and Engineering Managers

Your relationship with your EM must be build on solid ground. Your EM has to be a person you enjoy working with, you have a lot to learn from and a person you respect for the things he has done and still is doing. If you are not on the same page, it's less likely that you will benefit from this relationship as much as you could in a "good setup".
In my case, both of my EMs over the time were people I worked with from the first day in Productive. Over the years while working here, I learned a lot from them. The technical aspect is not as important here as the whole overview on how they get things done is - how do they approach problems and how do they solve them.
Other than having a good relationship with you SE/EMs, the main point here is that you have to trust their calls and support them publicly, to help them set those ideas in action. If you don't believe that what they are doing is going to help either you, your team or your organization, you will do a "bad job" here. That's because you in the first place won't accept those new ideas and implement them into your workflows and processes. How could we then expect the rest of our team to do so? One should lead by example.

Looking down the 🪜 - Software Engineers

On the other side, you also have to have really good relations with the people you "manage", the people in your team. In my case, being a TL for the Core Team, there are 3 Backend (BE) and 4 Frontend (FE) engineers (including me and the FE TL).
It is essential for us to know the strengths and weaknesses of our peers so that we could assign them the tasks in which they will prosper and in which they could learn and gain knowledge.
You also need to know them in order to know how to best approach them and to set your EMs ideas in motion. How to present those ideas to them in a way that they also see a benefit in them and not just "something new" they have to do.
These relationships and the work you make on this front will tell you if you are fit to be in management or not. Working with people is not for everyone and those who do it should do it for the right reasons - and that's basically to do it for the people you are managing and seeing them grow.

Looking at yourself 🪞

This is really important. Don't forget to look on yourself - on your motivation, on your moral and your overall "standards".
You could be great with both your SE/EM and your team members, but you shouldn't just be a proxy between them.
Have your opinions, question your EMs ideas, question your team members and look at things without bias. Not everything proposed by any side should be accepted blindly - as I said before, you should know all the details of the changes you are implementing so that you could expect people to follow.
For each idea that you are working on you should know what is the thing/problem your are solving, how are you solving it, what are the benefits of doing this and what could you come by down the road.
Those are the topics that you will most likely be going over with your EM.

What to take home from this 🙃

Ask for Feedback

What I think that people in this role, actually any new position, would benefit the most is feedback

For me, feedback is always welcome. Often times I even ask for it before the scheduled 1-on-1s, 360s or other kinds of performance reviews.

It should be pretty obvious when a newly promoted TL starts doing some of his new chores and that can go in both directions. Anyway it goes, it's great when your EM has your back. When he shares positive feedback about the good things you are doing and how to make the "bad" things better, how to handle them in a more appropriate way.

So my dear EMs, watch on your TLs, look out what they are doing, tailor your 1-on-1's to check the progress and their thoughts. Try remembering when you went thru this phase and what helped you to adjust to this new role.

Tell the people around you what's up

I noticed that my communication to some of my peers lately changed. In our chat there was a lot more work related messages - asking for status updates, asking for estimates, proposing them changes in their workflows, asking for explanations on decision, ... But I never actually told them why did I start making all those questions, asking for answers and started giving advice on workflows/processes.

So I wrote them a message explaining that now, as a TL and owner of some milestones, I have to have a better overview of their work and that we will have to communicate more often.
That I will probably be annoying them with status checks on tasks - trying not to micromanage tho.
That they can involve me in their Github Pull Requests (PRs) where they usually wouldn't - not as a reviewer, just as a subscriber.
I told them that I'm here to help them get unstuck with their task, at least try to help and to further delegate the issue.
That they can always ping me if they needed advice.
And I ended it with - "give me feedback at any point, correct me if doing something wrong or if you don't agree with me on something".

And when I start a conversation with someone new, this is now one of the first messages I start the conversation with.

That seems to be working well so far.

All in all

Ain't gonna lie, this really is a journey with a lot of downs and occasional ups. It's a change that sets you far out of your comfort zone. Just when you think you figured something out, you spot a flaw and find yourself two steps back.

Thoughts of going back to being an individual contributor sound great most the times you hit a low 😅 But if you remember how you learned any new thing, that probably looked a lot like this: you get out of your comfort zone, fail a few or a lot of times but eventually you figure it out. We must get out of our comfort zone to learn and to grow.

Ultimately, this trip to management is not irreversible, you can always take a step back and give it another shot later. Reflect to this experience and try to get the max out of it so that you know better next time.

So, does it suck to be in the middle?

If you ask me, depends on the day 😅 I would say no it doesn't, but damn, it sure isn't easy.

Watch Out When Overriding Memoized Methods

Nikola Buhinicek — Mon, 08 Jan 2024 09:34:33 +0000

This post was inspired by a 🐛 that was caused by not thinking enough when overriding a memoized method. Once we noticed the buggy behaviour, it took me some binding.pry time to figure out what was going on.

Lets check out what happened.

Code snippets

module Actions
  class FormAction
    attr_reader ...

    def initialize(...); ...; end

    def execute_action
      interpolate_properties

      form.process(form_attributes)
    end

    private

    def interpolate_properties
      interpolatable_attributes.each do |attribute|
        form_attributes[attribute] = interpolate(attribute)
      end
    end

    def interpolatable_attributes
      raise NotImplementedError
    end

    ...

    def form_attributes
      @form_attributes ||= action[:attributes].stringify_keys!
    end
  end  
end

A simple Ruby class which is the base class for the specific actions we need to implement for our Rails app.

The action for which we figured out the bug was the action of creating comments on various objects. Its code looks like:

module Actions
  class CreateComment < FormAction
    private

    def interpolatable_attributes
      ['body']
    end

    ...

    def form_attributes
      super.merge("#{commentable_type}_id": action_object.id)
    end
  end
end

Not much code is actually shown but just enough to spot the bug 🐛. Found it?

What was going on

Basically what was happening when running CreateComment#execute_action was the following:

In the execute_action we firstly need to interpolate some attributes - defined by the specific action class. In this case, for the CreateComment action that was only the 'body' attribute.

def execute_action
  # form_attributes = { 'body' => 'non interpolated' } 
  interpolate_properties

  form.process(form_attributes)
end

That's fine - we do some logic, the 'body' attribute gets interpolated and that value is set as the new value for the 'body' key in the form_attributes hash.

def execute_action
  interpolate_properties
  # form_attributes = { 'body' => 'interpolated' }
  form.process(form_attributes)
end

But, as the implementation of the CreateComment class uses super.merge(...) what we get is actually a new hash each time the method is called.

def execute_action
  interpolate_properties

  form.process(form_attributes) 
  # process method called with { 'body' => 'non interpolated' }
end

The next time, after the interpolation is over, the method form_attributes is called when passing that hash to our form object and in that moment, the form object gets a newly generated hash that doesn't have the interpolated 'body' value. So the interpolation was actually pointless as would any other modification of the form_attributes hash prior to the form.process call be.

The solution 🐛🔨

module Actions
  class CreateComment < FormAction
    private

    ...

    def form_attributes
      @form_attributes ||= super.merge("#{commentable_type}_id": action_object.id)
    end
  end
end

Added memoization to the CreateComment#form_attributes method.

Simple as that and when you think about it, also pretty obvious that it needed to look like this from the start.

p.s. These snippets maybe look a bit too simple and memoization probably looks like not needed but it's a stripped version of the real classes. The form_attributes method is called and modified quite a few times 🙃

A Close Call with Real-Time: How Rethinking Pub-Sub Saved the Day

Nikola Buhinicek — Fri, 15 Dec 2023 08:43:10 +0000

All of this started while I was working on our new feature - Automations 🤖. In a nutshell, Automations allow customers to set up actions triggered under specific conditions. Some of the currently implemented actions are sending slack messages, creating and updating tasks, or posting comments on objects.

I wanted that actions that modify tasks and comments would send a message over our real-time system so that our frontend clients (browser, mobile, desktop app) could pick that up and show those changes as they occur.

Currently, our app’s real-time updates are tied to POST/PATCH/DELETE requests. We have a controller extension Extensions::Broadcastable that hooks on save_form and destroy_resource methods and sends a real-time event if the action was successful. However, this approach wasn't suitable for my automation actions, as they don't go thru controllers.

module Api::V2::Extensions::Broadcastable
  extend ActiveSupport::Concern

  private

  def save_form
    super && broadcast_event(action_name == 'create' ? 'created' : 'updated')
  end

  def destroy_resource
    super && broadcast_event('deleted')
  end

  ...
end

As I was digging into this topic, I somehow changed the scope of my task from making the automations actions “live” to revamping our whole broadcasting architecture. I wanted to move that logic out of controllers to a place where I could catch all the changes - which would also catch the automations actions.

📖 Sidestory: A Recent Development in Pub-Sub

Just recently, Stef implemented a Publish-Subscribe architecture in our Rails app. He made it while revamping our search feature. That was a pretty cool moment for the team and it sounded really useful. After the presentation about it, we immediately started thinking what of our current code could've been done with Pub-Sub. No one really revamped anything by the time I was working on this real-time topic.

🧐 Exploring Different Approaches

1. Callbacks

Yeah... altho we all know that callbacks are evil, I couldn’t at least think about them, just for a moment...

2. Forms instead of Controllers

Both the controller actions and my automation actions use the same forms to handle our data. So, why wouldn't I hook on forms and send my real-time messages from there.
This approach was bugging me a bit as we don't use Form objects in all the places we are actually changing our data. So this wouldn't make the whole app feel "live" but I would cover more places than what we have currently. That sounded promising to me.

Wanted to pitch my thoughts to the rest of the Core team. Thru that discussion, a lightbulb moment happened 💡

3. Embracing Pub/Sub

I was mind-blown 🤯 That is exactly what I wanted!!

Publishing events happens for every change. Ofc, except the places we are using methods that skip callbacks (update_column, update_all, ...) as Pub/Sub and the aspect of publishing changes essentially is hooked to callbacks - but that’s a topic for itself 🫠

Making a PoC

As with all big changes in our codebase, and generally in our product, I was putting this code behind a Feature Flag (FF).
Simply, when one would have the pubSubBroadcasting FF enabled I would skip sending the real-time events from the controller actions and I would handle the published events accordingly. If you didn't have that FF, nothing changed.

class Api::V2::Tenanted::TasksController < ApiController
  include Extenstensions::Broadcastable

  ...

  def should_broadcast?
    return false if FeatureFlags.enabled?('pub-sub-broadcasting')

    super
  end

  ...
end

Made a few Subscribers that would listen for all the task and comment related changes and simply handle them as sending a real-time event.

class Core::Tasks::Broadcaster < Realtime::Broadcaster
  PubSub.subscribe('task.upsert', self, :on_upsert)
  PubSub.subscribe('task.delete', self, :on_delete)
  PubSub.subscribe('comment.upsert', self, :on_comment_upsert)
  PubSub.subscribe('comment.delete', self, :on_comment_upsert)

  ...
end

And ofc, added RSpec tests and set up some widgets on New Relic to cover the difference in number of events that we are sending now - as we knew that we are going to send more events now.

Basically that was it. The next step was to slowly propagate this FF over all organizations, check our New Relic metrics and see if nothing breaks. Once we would release that change to all of our user base, we should cover the remaining objects and make subscribers for their Pub/Sub events too.

Showing off with it

As I was pretty proud of this solution, I kinda talked a lot about it and so naturally it popped up in a 1on1 meeting with my EM Lucin. He wanted to discuss that a bit more so I told him the same thing I wrote here. My vibe was like "Isn't that great? Our APP will be LIVE, all the data would be in sync."

His response was "But do we really want that?".

That wasn't the response that I was looking for 🙃

What I didn't know was that our frontend client, once it gets socket messages, depending on the screen the user is, has to make additional API calls so that all the required data could be fetched again. So, a great deal of my real-time events actually ends up generating additional requests to our server and in a way, we are just generating a lot more traffic (self DDoSing?). As I didn't know about this, I wasn't even paying attention to those metrics along the way.

One step forward, two steps back

Let's get better data so that we can make a better call on this.

I took a period of one week from our logs and checked the number of the POST/PATCH/DELETE requests versus the number of dispatched publish events. This in the end would roughly be the same number as the events we are sending over the socket in the current and in the new way.

Tasks endpoint
controller POST/PATCH/DELETE actions - 211k
task.upsert + task.delete publishes  - 263k
> thats ~25% more real-time events

Companies endpoint
controller POST/PATCH/DELETE actions      -  5.2k
company.upsert + company.delete publishes - 20.1k
> thats ~4 times more real-time events

Deals endpoint
controller POST/PATCH/DELETE actions -    31k
deal.upsert + deal.delete publishes  - 1_728k
> thats ~55 times more real-time events!!!

I wasn't really aware of how badly this could end up. I was making a PoC out of this for tasks and comments and you can see here that there wasn't such a big difference. 25% more events was okay, I knew it would be more.

But look at our deals endpoint for example - 55 times more events would be sent. That would add up to the traffic we sent over sockets, to our infrastructure bill for that services, and I don't want to imagine the number of API calls generated as a result of this - by ourselves...

Deals have that much traffic because a lot of other objects update financial data on deals so that was understandable too, it immediately came to us...

Back to the drawing board

1. Pub/Sub shouldn't be a bad call for this

As this is the part of our code that gets all the changes in our data, when wanting to make a frontend client to be as live as it gets, this should be a good call. The solution would be in not sending all the changes over sockets but filter them by relevant and not relevant. This way we would surely see a drop of those mad company and deal numbers.

2. Send all the needed data to front?

As mentioned before, each real-time message already contains the object that was changed. The issue here is that there are a lot of screens in our client and we can't know all the contexts our users are in and what additional data should be sent - which is exactly why our client makes API calls when receiving some socket messages.

3. Why didn't I just resolve the problem I had

To make those actions "live" in our frontend clients (browser, mobile, desktop APP), I wanted to plug them into our real-time system.
So, why didn't I just put a bit of explicit calls to the code of my automations actions where I would just call the class that sends the real-time message.
But no, I wanted to play smart and to fix a problem that wasn't really there in the first place - to make everything more live while no one was asking for it.

The Aftermath

So yeah, the Pub/Sub usage in our real-time part of the app is put on pause until we leverage things up.

I went on with the 3rd solution and added 5 lines of code - a call to the Broadcaster class is one line and I have 5 events over 4 classes to send...

This was a nice learning opportunity for me and I would say that:

When having concrete problems, stick to handling them and resolve them first
I had the data the whole time in New Relic, I should've prepare better
It's not bad to be explicit in code, not everything should be an abstraction, generalization, metaprogramming, ... Hope to write on this point a bit more soon

Good thing in this story is that we didn't actually do any damage with this and we didn't lose a lot of time.

Anyone faced similar problems? If yes, how did you deal with them?

Custom Fields: Give Your Customers the Fields They Need

Nikola Buhinicek — Wed, 06 Dec 2023 10:20:00 +0000

Here at Productive, we’re building an operating system for digital agencies.

But, because each agency is different (think type, size, services they offer, the way they’re set up as an organization…), they need customization options for their workflows. So it’s pretty hard to model all those needs and use cases through a unified data model.

If only there were a way to let them shape those models to their own needs.

Let’s say that one of our customers, ACME digital agency wants to keep track of their employees’ nicknames and to be able to search them by that field. Other than that, they would also like to keep track of their birthdays and be able to sort them and group them by that date.

To me, as a developer, this sounds as simple as it gets—add two new columns to the people table, open those attributes to be editable over the API and send them back in the response.
But should we do that? Should we add all kinds of fields to our models even if those fields are going to be used only by a handful of our customers?

Let me show you how we tackled this type of feature request and made a pretty generic system around it.

What Did Our Customers Want?

It was pretty clear to us what our customers wanted, and that was:

to be able to add additional fields to some of our models (People, Projects, Tasks, …)
to have various data types on those fields (text, number, or date)
to be able to search, sort, or even group by those fields

Our Approach

The Custom Field Model

As we’re building a RESTful API that’s formatted by the JSON:API specification and store our data in a MySQL8 relational database, a few things were pretty straightforward – we need a new model and we’ll name it Custom Field (naming wasn’t an issue here 🥲).

The main attributes of that model should be:

name - string
description - string
data_type_id - integer (1 - text, 2 - number, 3 - date)
  -> enumeration that described the data type of the field
customizable_type - string 
  -> tells us on what data model will that custom field be available, eg. Person, Project, Task, ...
  -> could've also been an enumeration 😅

How To Store the Field Values?

OK, so now that we know how to define custom fields, how can we know which value someone assigned to a custom field for some object? And where to store that information?

Three possible solutions came to mind:

1. Add a limited number of custom_field columns to our models

We can add a few custom_field columns to our models and that will work for some of our customers but there will always be others that need few extra fields. Adding numerous columns to our models surely isn’t the best solution, we can do better than this 😅

2. Add a join table

As mentioned before, while relying on a relational database, a join table sounds like the go-to approach. That table would be a simple join table between the custom field and a polymorphic target (yay, Rails 🥳). Other than those foreign keys, we would have a column to store the value.

3. Add a single JSON column to our models

This sounded as flexible as it gets. It would be a simple map where the key would be the custom field ID and the value would be the assigned value for that custom field.

Why We Ended Up Choosing JSON

The first solution was just too limited so we discarded that one immediately and focused on the remaining two solutions.

On one hand, a better design would be to have the custom field values represented by a model but on the other hand, we won’t actually do much with that data. That would just be data that our users set on our objects, data that isn’t important for our business logic. So a simple JSON column didn’t sound bad either.

The searching and sorting aspect of this feature request was probably the most important one for us. That was supposed to work as fast as it gets, without being a burden to our performance.

That’s why we implemented both solutions, tested a lot of searching/sorting/grouping scenarios (we’ll go through that in more detail soon), and then checked the metrics.

The faster solution was the second one, the one with the JSON column, and that made sense to us. That solution doesn’t use JOIN clauses in SQL since the values are written directly in the searched table and can be queried in the WHERE clause. Luckily for us, MySQL8 supports a bunch of great functions to work with JSON columns (JSON_EXTRACT, JSON_UNQUOTE, JSON_CONTAINS and others).

Great! Now that we know how to store the custom field values too, let’s dig into the coding.

From a development point of view, we did the following:

Added a new model, Custom Field, and implemented CRUD operations that can be called over the API
Wrote schema migrations that added a JSON column – custom_fields – to some of our models (people, projects, tasks, …)
Opened the custom_fields attribute so it can be edited over the API
Wrote a generic validation that checks if all the values in the custom_fields hash have the appropriate data type
Added the custom_fields attribute to the API response of the appropriate models

That was most of the work we needed to do to be able to manage custom fields in our models.

But…what about the searching and sorting aspect of custom fields?

Searching Through Custom Field Values

We already had a generic solution written for searching over the API.

We have a format of sending query params for searching, like filter[attribute][operation]=value. For searching through custom fields, we wanted to keep the same format so we ended with a quite similar one – filter[custom_fields][custom_field_id][operation]=value.

We had to add an if-else statement that would handle the custom fields filtering in a different way than filtering through other attributes as the format contained one additional argument — custom_field_id.

What was different in the filtering logic was that we have to load the custom field that’s being filtered by and check what data type its values are. That’s needed to cast the values into numbers or dates—text values don’t make a difference.

So the query params and its SQL query counterparts, based on custom field type, would look like this:

/* for a text custom field and query filter[custom_fields][1][eq]=abc */
LOWER(custom_fields->>'$."1"') LIKE '%abc%'; 

/* for a number custom field and query filter[custom_fields][2][eq]=42 */
CAST(LOWER(custom_fields->>'$."2"') AS DECIMAL(20,10)) = 42; 

/* for a date custom field and query filter[custom_fields][3][eq]=2022-11-01 */
DATE(LOWER(custom_fields->>'$."3"')) = '2022-11-01';

/* The ->> operator is an alias for JSON_UNQUOTE(JSON_EXTRACT(...)) */

Sorting by Custom Field Values

The concept of sorting by attributes is something we also already tackled by abstracting logic.

The only thing that changes when sorting by custom fields is that we first need to cast the values and then sort by them.

Once again, there’s a small change in the format for custom fields sorters (sort=custom_fields[custom_field_id]) compared to when sorting by a standard attribute (sort=attribute). We need to handle the custom_fields sorters separately because we have to load the desired custom_field and check its type.

Then the ORDER BY statement, based on custom field types, looks like this:

/* for a text custom field and query sort=custom_fields[1] */
ORDER BY CAST(custom_fields->>'$."1"' AS CHAR) ASC; 

/* for a number custom field and query sort=-custom_fields[2]
Noticed the - on the begining? That defines the order of sorting - DESC or ASC */
ORDER BY CAST(custom_fields->>'$."2"' AS unsigned) DESC;

/* for a date custom field and query sort=custom_fields[3] */
ORDER BY DATE(custom_fields->>'$."3"') ASC;

Grouping by Custom Field Values

This was a fun one. The main point here was that you should include the custom fields as some kind of columns stated in the SELECT statement so that you could later use those columns in the GROUP BY statement.

To get the custom field in the SELECT statement, you have to create a virtual column for it. All we needed to do was to extract the values of the grouped custom field and give that virtual column an alias so that we could reference it in the GROUP BY statement. For the column alias we went with the format custom_fields_{custom_field_id}.

For a custom field with id=x, this is done as following:

SELECT ..., custom_fields->>'$."x"' AS custom_fields_x

Once we have the virtual column defined, the grouping part gets done simply, by adding the GROUP BY statement with the earlier mentioned alias.

So in the end, you get a SQL query like:

/* query group=custom_fields[1] */
SELECT ..., custom_fields->>'$."1"' AS custom_fields_1
FROM ...
WHERE ...
GROUP BY custom_fields_1

What Our Customers Got

A simple way to define Custom Fields:

And a place to assign values to their fields:

Summa Summarum

We made it possible for our customers to define custom fields in our data models. Also, we made it possible to search, sort and group by those fields.
It wasn’t long before we had even more requests that built upon our custom fields architecture. The fields we supported at first were okay, but now our customers wanted more field types. They wanted:

to have dropdown custom fields
to have relational custom fields
a field where the values would be objects from one of our existing data models

But before we dig into that, let’s give some time for this basics to sink in. I’ll be back soon with another blog post in which I cover how we solved that new set of feature requests

Here to stay - Analyzing RubyGems stats for 2018

Nikola Buhinicek — Tue, 22 Jan 2019 19:10:05 +0000

As years go by, there is always a question of whether Ruby is dead or alive, and as usual, we have done our research and compiled some statistics about Ruby’s 2018 lifeline that might help you find an answer.

2018 was an interesting year in terms of gems. Some increased, and some decreased their download numbers. Some were well maintained, while others could have used a helping hand. We have also seen a few newcomers that almost made it to the top of their categories. As for the language itself, at the end of 2018, Ruby 2.6 was released, getting us a few steps closer to achieving the Ruby 3×3 goals. The most notable improvement in this release was the brand new (experimental) JIT compiler that should boost the performance of Ruby programs.

Sources

The data used for generating the statistics below were obtained from two sources – the latest RubyGems PG-dump and BestGems. The latter keeps track of all daily gem downloads and exposes this data through their API. Having that and Vedran’s awesome script, which made it pretty easy to browse through the numbers, we have compared some of the most used gems.

Bear in mind that some of these downloads are a direct result of gem dependencies, non-cached CI builds, and gem mirroring services. There is still a strong correlation between the number of gem downloads and the number of its active users, but we feel it’s important to make it clear that these two numbers are not exactly the same.

Gem creation & releases

Unfortunately, this was another year in a row that the creation of new gems decreased even further. That’s not so surprising after all because there already is a huge number of gems that are well-known, well-maintained, and cover a lot of common use cases. To make a new gem in a field that’s already well covered is a real quest, but not impossible. Further down, we will introduce a gem that managed to make a huge dent in the JSON:API field.

As for the creation of gems, a downhill trend is also evident in the number of gem releases. But you can’t get one without the other. Older and well-maintained gems usually get a few new minor releases and a major release in a calendar year. Newer gems that are still in infancy usually have more code churn, breaking changes, and consequently releases.

Rails downloads

In 2018, the number of Rails downloads had increased again. It had its ups and downs, but it held around the 3 million download mark. As the end of the year came, we heard a lot about the new major release, Rails 6. People jumped on the hype train raising the number of monthly downloads to almost 4 million.

Action Mailbox, Action Text, and parallel testing are the most anticipated additions, but there will be more improvements included in Rails 6. We will need to be patient though, as the release is scheduled for the end of April 2019.

Other web frameworks

Sinatra – still falling behind

When Sinatra got a new maintainer in 2017, and they pushed a new 2.0 release, things started to look better and the upward trend continued into 2018 as well. Having just five patches throughout the year, they reached 1,5 million downloads as their monthly standard, pushing it to the all-time high of just above 2 million downloads at the end of 2018.

On the other side of the download spectre, Padrino yearly downloads fell by 10%, while Hanami downloads increased by about 33%. Hanami’s core team is currently working on version 2.0, which could be something to look out for in 2019. Their download numbers are much lower compared to Rails and Sinatra but that doesn’t detract from the fact that there are people that find them more suitable for their use cases and see potential in their further development. Having options to choose from is always a positive thing.

Database adapters

For PG and MySQL2 2018 was a great year. PG managed to increase their numbers for 4, and MySQL2 for 5 million downloads. SQLite3, on the other hand, experienced a drastic drop to only one-third of the downloads made in 2017 (from 21 to 8 million yearly downloads).

Application servers

The situation here is pretty much the same as the year before. Puma, the default server for Rails, is still in the number one spot. And Thin, with no new releases in the past year, actually made a 250 thousand drop.

Unicorn, although better maintained than Thin, didn’t manage to increase the number of downloads compared to 2017. Developers behind Passenger, on the other side, are putting a lot of work into new releases, having a 6.0 version released at the end of November and managing to triple their download numbers since 2017. They had a huge spike in the middle of the year, recording more downloads in one month than in the last two years combined.

Other ORMs

2018 maybe wasn’t the best year for Sequel, as its downloads dropped by 14%. Sequel is well maintained and has a status that just a few gems have – 0 outstanding issues. Other than that, we could mention rom-rb, which is maybe not a strict ORM, but has the same uses. It had a small growth during the year, recording just over 90 thousand downloads.

Testing

For RSpec it seems that only the sky’s the limit. Its downloads more than doubled hitting almost 200 million. Even though it’s defined as a development dependency in a lot of gems, it’s not used as a runtime dependency that often. The vast majority of the downloads thus stem from actual project uses. Minitest, on the other hand, gained a bit more popularity, although there are some open issues and the last release was just under 12 months ago. All in all, it’s great to see these numbers skyrocket because that just means that people are testing their programs more and that the testing culture is evolving even further.

Debugging

It is inevitable that bugs will occur during software development, but we can examine them easily using the following gems. The downloads of pry, irb’s number one enemy, jumped by 50%, but this number is eclipsed by the number of open issues – 177 as of writing this article. Byebug, which is just below pry, has a download number that is growing even faster, presumably because they started 2018 by releasing version 10.0.

Another interesting debugging tool – Better Errors. They replaced the standard Rails error page with their own, where except the usual stacktrace, you can also debug your code as you were in a debugger. It had a consistent growth rate that continued into 2018 as well. Well maintained and released, it is a very helpful gem to have up your sleeve.

Background workers

The situation here is the same as the year before. Sidekiq is way ahead of Delayed::Job and Resque, by 10 million downloads.. By removing even more dependencies and constantly improving and releasing, Sidekiq has become much faster and leaner than its competitors. Resque dropped a 2.0 release in November, but there are no spikes visible on our graph. Comparing it to Delayed::Job, their numbers matched each other throughout the year.

Authorization

The authorization has been pretty stable too. CanCanCan is still in the lead, while Pundit came in second with 7 million downloads less. While CanCanCan’s downloads were falling through the year, Pundit’s, on the other hand, were stable until July and then started to increase. The reason for that is the 2.0 release, which Pundit fans had been waiting for more than two years. Pundit could take the cake and dethrone CanCanCan if the download trend continues into 2019.

Administration

It was a big year for Active Admin, its downloads doubled and it left Rails Admin far behind. Both are still well maintained and follow similar release cycles.

JSON:API Specifications

This is a big one. The most popular and talked-about gem in the JSON:API world, Fast JSON API by Netflix, which was first released in February 2018, surpassed JSONAPI-rb on a yearly download base, and it could soon become THE gem to use for JSON:API serialization. JSONAPI-rb missed a whole year in terms of releases, and at JSONAPI::Resources, the issues slowly piling up. Having that in mind, it’s no wonder that there are ever more migrations to the newcomer, that is well maintained and cared for.

File uploaders

2018 wasn’t the best year for CarrierWave and it was even worse for Paperclip, which was deprecated. Both were well maintained, and released often, but they managed to get just 20-30% more downloads than in 2017. Finishing the year as equals and not actually growing that much are signs that something’s off.

The reason for that is just what we predicted last year as a possible game-changer – Active Storage. Available from Rails 5.2 onwards, it gained a lot of popularity and finished the year with 5 million downloads.

Shrine is another great solution for file upload management. While the numbers seem low, having been downloaded just over 300 thousand times, it offers a lot of advantages over the more popular choices, with generality, simplicity, and modularity being at the forefront.

Aggregated stats for 2018

The number of Rails downloads continued to grow and reached 37,5 million. 29 new gems were published daily on average. 260 new gem releases were published daily on average. Taking into consideration just the gems mentioned in this post, total downloads jumped from 389,794,028 in 2017 to 499,035,801 in 2018. That’s a 28% increase in just one year!

Conclusion

As in the previous years, 2018 was another successful year for Ruby, and if after all of these numbers, the state of Ruby is not clear to you, let us assure you that it is here to stay. We are not the only ones that came to this conclusion.

With more than 3,700 contributors and new ones coming in on a daily basis, Rails is way ahead of, for example, Django, which has just over 1,600 contributors. That number of Rails backers just proves that Ruby has a strong community that is involved in Rails development and makes a lot of gems that in the end, help other developers build awesome applications better and faster.

Maybe Ruby isn’t the number one language to pick in terms of performance, as there are many more performant choices. But it is clear that performance is of great importance with the latest Ruby releases just by looking at the GC and JIT changes done by the Ruby Core team. Achieving the 3×3 promise (Ruby 3.0 being three times faster 2.0) thus seems closer by the day.

So, buckle up, as there are a lot of things about to change. And in the meanwhile, if you want to dig into the data feel free to download it here.