DEV Community: Evil Martians

Nano Stores in Angular: how to make the state management simpler

Nina Torgunakova — Fri, 21 Apr 2023 15:05:15 +0000

TLDR: Angular now has integration for Nano Stores, an open source state manager based on the idea of atomic tree-shakable stores and direct manipulation. It is very small (from 334 bytes), has zero dependencies, and promotes moving logic from components to stores. With such stores, you don't need to call the selector function for all components on every store change, which makes them noticeably fast.

Here is a small example of how we can use Nano Stores in an Angular component:

import { Component } from '@angular/core';
import { atom } from 'nanostores';
import { NanostoresService } from '@nanostores/angular';
import { Observable, switchMap } from 'rxjs';

const user$ = atom<{ id: string; name: string; }>(
  { id: '0', name: 'John'}
);

@Component({
  selector: "app-root",
  template: '<p *ngIf="(currentUser$ | async) as user">{{ user.name }}</p>'
})
export class AppComponent {
  currentUser$: Observable<IUser> = this.nanostores.useStore(user$);

  constructor(private nanostores: NanostoresService) { }
};

How can the appearance of Nano Stores in Angular improve our developer experience with state management? Let's start from the beginning.

Nowadays the concept of state management is crucial for client-side development: we need a reliable source of truth to manage data in our applications.

And in Angular, there exist specific approaches to manage state. In this area, there are a lot of implementation details that we’ve all become accustomed to over the years.

What do we have now?

When developing projects with Angular, we usually manage data state in the services, not in components or modules. But, using this approach, when the number of features increases, the number of services also increases and management becomes increasingly complicated. Because of that, we need a convenient approach to handle this process, which should also be predictable in order to reduce the number of possible bugs.

Angular developers have typically been used to implementing state management logic using RxJS or concepts like NgRx/NGXS inspired by Redux, because these were the most popular approaches. Let's try to write a simple application with state management using these instruments and understand if they are really convenient to use.

RxJS

Inside Angular, we already have a built-in option: RxJS (Reactive Extensions for JavaScript), a library for reactive programming, remains the most popular solution to compose callback-based or asynchronous code. We can also handle changing the state of our project by using RxJS operators, and it is quite a common solution, and isn’t too bad for many cases.

Let's build a familiar example: a TODO List application. We’ll consider three functions for the list of tasks: adding a new task, deleting an existing task, and toggling the status of the task from undone to done, or vice versa.

Here’s our HTML:

<input #newTask/>
<button (click)="addTask(newTask.value)">Add</button>
<ul class="todo-list" *ngIf="(tasks$ | async) as tasks">
  <li *ngFor="let task of tasks">
    <button (click)="toggleStatus(this.task.id)">Toggle Status</button>
    <span [ngClass]="{'done': this.task.isDone }">{{this.task.name}}</span>
    <button (click)="deleteTask(this.task.id)">Delete</button>
  </li>
</ul>

Then, we can define a service with basic methods for state management (which we can extend in the future with a more detailed implementation):

import { BehaviorSubject, distinctUntilChanged, map, Observable } from 'rxjs';

// This type will be passed when extending the StateService
export class StateService<T> {
  // This field stores the current state's value and emits it to the new subscribers
  protected state$: BehaviorSubject<T>;

  // Initializes the state$ with some initial value
  constructor(initialState: T) {
    this.state$ = new BehaviorSubject<T>(initialState);
  }

  // Returns the current value from the state
  get state(): T {
    return this.state$.getValue();
  }

  // Emits the new state value
  setState(newState: Partial<T>) {
    this.state$.next({...this.state,...newState });
  }

  /**  Called when state$ emits a new value.
  Operator distinctUntilChanged() will skip emissions
  until the piece of state obtain a new value or object reference */
  select<K>(mapFn: (state: T) => K): Observable<K> {
    return this.state$.asObservable().pipe(
      map((state: T) => mapFn(state)),
      distinctUntilChanged()
    );
  }
}

Notice that there are no mentions of the actual TODO tasks in this part of the code. So, you can use this service later for new features that are not related to the TODO list.

Now we can extend StateService with a more practical service, TodosStateService:

@Injectable()
export class TodosStateService extends StateService<{ tasks: Task[] }> {
  /** In the constructor we call the constructor of StateService and pass the initial state (empty array).
   * Injected instance of service with method getTasks() expose an API to receive the tasks. */
  constructor(service: TodoListService) {
    super({ tasks: [] });
    service.getTasks().subscribe(tasks => this.state$.next({ tasks }));
  }

  getTasks(): Task[] {
    return this.state.tasks;
  }

 // Exposes the corresponding state data to subscribers
  todos$: Observable<Task[]> = this.select(state => state.tasks);
}

Here’s the structure for a TODO list Task:

type Task = {
  id: string;
  name: string;
  isDone: boolean;
}

In the todoList.component.ts we need to inject instances of two services: TodosStateService and TodoListService.
TodoListService is a service that calls API methods for receiving, adding, deleting, and changing tasks:

constructor(
    private todosStateService: TodosStateService,
    private service: TodoListService,
  ) { }

And we need to subscribe to $todos in the TodosStateService. We can do this by declaring the component field like this:

tasks$: Observable<Task[]> = this.todosStateService.todos$;

But don't forget to unsubscribe from this subscription when the component has been destroyed to prevent possible memory leaks. (You can do this any way you prefer, for example using RxJS operator takeUntil.)

Finally, let's implement actions with our tasks in the list: adding, deleting, and status toggling. I also suggest trying to implement this using the optimistic UI pattern: we’ll update the UI even before receiving a server response:

addTask(name: string) {
    const newTask = {id: '...', name, isDone: false};
    this.todosStateService.setState({tasks: [...this.todosStateService.getTasks(), newTask]});
    this.service.addTask(newTask);
}

deleteTask(id: string) {
    this.todosStateService.setState({
        tasks: this.todosStateService.getTasks().filter(task => task.id !== id)
    });
    this.service.deleteTask(id);
}

toggleStatus(id: string) {
    this.todosStateService.setState({ tasks: this.todosStateService.getTasks()
        .map(task => ({...task, isDone: task.id === id ? !task.isDone : task.isDone }))
    });
    this.service.toggleStatus(id);
}

These functions will invoke the setState method in the service that maintains the state of tasks.

That's all for the RxJS approach; it’s scalable but the biggest drawback is the large number of routine tasks. We need to create an entirely new service for the whole app, then create a new one that extends it each time when we need a feature with new entities.

NgRx

Let's next try to develop the same state management example with NgRx, a very popular state management library for Angular. It was inspired by Redux and based on such concepts as Stores, Actions, Effects, and Reducers.

First of all, we’ll create a new file with actions. In NgRx, Actions are unique events that happen throughout the application. For simplicity in this case, we can define actions for a single task (adding, deleting, toggling status), and also the action to receive all tasks from the server (we can do it also by Effects, but let's stick with the easiest approach for this small example):

import { createAction, props } from '@ngrx/store';

export const LoadTasks = createAction('Load Tasks', props<{tasks: Task[]}>());
export const Add = createAction('Add', props<{task: Task}>());
export const Delete = createAction('Delete', props<{id: string;}>());
export const ToggleStatus = createAction(Toggle Status', props<{id: string;}>());

And we need the Reducer for the tasks that will manage the state of our tasks:

import { createReducer, on } from '@ngrx/store';

export const taskReducer = createReducer([],
  on(LoadTasks, (_, action) => (action.tasks)),
  on(Add, (state, action) => ([...state, action.task])),
  on(Delete, (state, action) => state.filter(task => task.id !== action.id)),
  on(ToggleStatus, (state, action) => state.map(task => ({...task, isDone: task.id === action.id ? !task.isDone : task.isDone }))),
);

(Did you notice we already have a lot of boilerplate for simple things?)

We need to add the StoreModule.forRoot function in the imports array of NgModule and pass the object that contains the taskReducer. The StoreModule.forRoot() method registers the providers needed to access the global store throughout the application.

import { StoreModule } from '@ngrx/store';
imports: [
    StoreModule.forRoot({todoState: taskReducer}),
    ...
],

Well, now we can create the component with tasks. We have exactly the same HTML that we had for RxJS example, but the code in component.ts will be different.

After creating the new component, we need to inject the instance of TodoListService (we already used this in the previous example), and the Store service to dispatch the task actions:

constructor(private service: TodoListService, private store: Store<{ todoState: Array<Task> }>) { }

Then, we can subscribe to task changes in the ngOnInit method to receive the tasks from the API:

ngOnInit() {
    this.service.getTasks().subscribe(tasks => todos$.set(tasks));
}

Keep in mind that, in this case, we also probably want to unsubscribe from the subscription when the component has been destroyed.

We’re nearly finished. Let's add a field to the component with the actual state of tasks. We need to select them from the store using the select method we recently created:

tasks$: Observable<Task[]> = this.store.select(state => state.todoState);

With regards to Optimistic UI, all we need to do is to add the following code for the actions with tasks:

addTask(name: string) {
    const newTask = {id: '...', name, isDone: false};
    todos$.set([...todos$.get(), newTask]);
    this.service.addTask(newTask);
}

deleteTask(id: string) {
    todos$.set(todos$.get().filter(task => task.id !== id));
    this.service.deleteTask(id);
}

toggleStatus(id: string) {
    todos$.set(todos$.get().map(
        task => ({...task, isDone: task.id === id ? !task.isDone : task.isDone })
    ));
    this.service.toggleStatus(id);
}

All in all, we have changes in four different files, module.ts, component.ts, actions.ts, and reducer.ts (and we haven't even used Effects for even more simplicity.) We already have a lot of repeating actions to build just the basic features, imagine the result if we needed to add extended ones.

Other than that, NgRx is a heavy library for your bundle and we need to spend time to properly understand how all these concepts, like Actions, Reducers, Selectors, and so on, work.

But we can try another option to manage the state and make our life simpler.

Nano Stores

Let's try the same example with the TODO List application to see how to use Nanostores in this simple case — and how convenient it can be.

First of all, we need to provide injection tokens in our NgModule to make the Nano Stores work:

import { NANOSTORES, NanostoresService } from '@nanostores/angular';

@NgModule({ providers: [{ provide: NANOSTORES, useClass: NanostoresService }], ... })

Then we need to create a new atomic store to hold the tasks. The initial state for the tasks will be the empty array:

import { atom } from 'nanostores';
const todos$ = atom<Task[]>([]);

And here we also can define functions that we will invoke to update the value in our store:

export const addTask = (newTask: Task) => {
 todos$.set([...todos$.get(), newTask]);
}

export const deleteTask = (id: string) => {
 todos$.set(todos$.get().filter(task => task.id !== id));
}

export const toggleStatus = (id: string) => {
 todos$.set(todos$.get().map(
   task => ({...task, isDone: task.id === id ? !task.isDone : task.isDone })
 ));
}

It's time to create the component with tasks (the HTML is all the same as in the previous section). We will inject the instance of our usual TodoListService, and also the instance of NanostoresService:

constructor(private service: TodoListService, private nanostores: NanostoresService) { }

Now we can add a field into the component with the actual state of tasks using the useStore method in the injected instance of NanostoresService:

tasks$: Observable<Task[]> = this.nanostores.useStore(todos$);

And then, in ngOnInit, we can subscribe to the task changes method to receive the tasks from the server side and set them to the store:

ngOnInit() {
    this.service.getTasks().subscribe(tasks => todos$.set(tasks));
}

All that is left to do is to write the part of the component where we’ll call the functions that interact with the store (which we already wrote), and invoke the API methods:

addTask(name: string) {
    const newTask = {id: '...', name, isDone: false};
    addTask(newTask);
    this.service.addTask(newTask);
}

deleteTask(id: string) {
    deleteTask(id);
    this.service.deleteTask(id);
}

toggleStatus(id: string) {
    toggleStatus(id);
    this.service.toggleStatus(id);
}

And that's all. No additional files, utility services, and no boilerplate. Don't forget to unsubscribe from the initial subscription, as was described in previous sections.

Why is Nanostores the handiest way to manage state?

After solving this simple task with three different approaches and seeing the brevity and beauty of Nanostores, let's sum up the results. Why is Nanostores the best option to manage the state of an application in Angular?

It was designed to move logic from components to stores – less logic in components, clearer and more predictable development process
No boilerplate, short and clear syntax
It’s much smaller than NgRx and its analogs. For instance, even the base NgRx package @ngrx/store, has a minified bundle size 3 times more than nanostores and its integration combined (using the bundlephobia metric) No complex concepts like Reducers, Effects, or Selectors, which must be deeply understood before using. You just need to deal with small atomic stores
Nanostores has a growing community and its own developing ecosystem: we already have a tiny router for stores, a persistent store, and a tiny data fetcher

I believe that with open source projects like this, Angular can soon get rid of boilerplate, complex definitions, and huge packages. Try using Nanostores in your Angular project to check out all these advantages for yourself.

Climbing Steep hills, or adopting Ruby 3 types

Vladimir Dementyev — Fri, 11 Dec 2020 07:53:55 +0000

With Ruby 3.0 just around the corner, let's take a look at one of the highlights of the upcoming release: Ruby Type Signatures. Yes, types come to our favourite dynamic language—let's see what could work out of that!

It is not the first time I'm writing about types for Ruby: more than a year ago, I tasted Sorbet and shared my experience in the Martian Chronicles. At the end of the post, I promised to give another Ruby type checker a try: Steep. So, here I am, paying my debts!

I'd highly recommend taking a look at the "Sorbetting a gem" post first since I will refer to it multiple times today.

RBS in a nutshell

RBS is a language to describe the structure of Ruby programs (from Readme). The "structure" includes class and method signatures, type definitions, etc.

Since it's a separate language, not Ruby, separate .rbs files are used to store typings.

Let's jump right into an example:

# martian.rb
class Martian < Alien
  def initialize(name, evil: false)
    super(name)
    @evil = evil
  end

  def evil?
    @evil
  end
end

# martian.rbs
class Alien
  attr_reader name : String

  def initialize : (name: String) -> void
end

class Martian < Alien
  @evil : bool

  def initialize : (name: String, ?evil: bool) -> void
  def evil? : () -> bool
end

The signature looks pretty similar to the class definition itself, except that we have types specified for arguments, methods, and instance variables. So far, looks pretty Ruby-ish. However, RBS has some entities which are missing in Ruby, for example, interfaces. We're gonna see some examples later.

RBS itself doesn't provide any functionality to perform type checking*; it's just a language, remember? That's where Steep comes into a stage.

* Actually, that's not 100% true; there is runtime type checking mode. Continue reading to learn more.

In the rest of the article, I will describe the process of adding RBS and Steep to Rubanok (the same project as I used in the Sorbet example, though the more recent version).

Getting started with RBS

It could be hard to figure out how to start adding types to an existing project. Hopefully, RBS provides a way to generate a types scaffold for your code.

RBS comes with a CLI tool (rbs) which has a bunch of commands, but we're interested only in the prototype:

$ rbs prototype -h
Usage: rbs prototype [generator...] [args...]

Generate prototype of RBS files.
Supported generators are rb, rbi, runtime.

Examples:

  $ rbs prototype rb foo.rb
  $ rbs prototype rbi foo.rbi
  $ rbs prototype runtime String

The description is pretty self-explanatory; let's try it:

$ rbs prototype rb lib/**/*.rb

# Rubanok provides a DSL ... (all the comments from the source file) 
module Rubanok
  attr_accessor ignore_empty_values: untyped
  attr_accessor fail_when_no_matches: untyped
end

module Rubanok
  class Rule
    # :nodoc:
    UNDEFINED: untyped

    attr_reader fields: untyped
    attr_reader activate_on: untyped
    attr_reader activate_always: untyped
    attr_reader ignore_empty_values: untyped
    attr_reader filter_with: untyped

    def initialize: (untyped fields, ?activate_on: untyped activate_on, ?activate_always: bool activate_always, ?ignore_empty_values: untyped ignore_empty_values, ?filter_with: untyped? filter_with) -> untyped
    def project: (untyped params) -> untyped
    def applicable?: (untyped params) -> (::TrueClass | untyped)
    def to_method_name: () -> untyped

    private

    def build_method_name: () -> ::String
    def fetch_value: (untyped params, untyped field) -> untyped
    def empty?: (untyped val) -> (::FalseClass | untyped)
  end
end

# <truncated>

The first option (prototype rb) generates a signature for all the entities specified in the file (or files) you pass using static analysis (more precisely, via parsing the source code and analyzing ASTs).

This command streams to the standard output all the found typings. To save the output, one can use redirection:

rbs prototype rb lib/**/*.rb > sig/rubanok.rbs

I'd prefer to mirror signature files to source files (i.e., have multiple files). We can achieve this with some knowledge of Unix:

find lib -name \*.rb -print | cut -sd / -f 2- | xargs -I{} bash -c 'export file={}; export target=sig/$file; mkdir -p ${target%/*}; rbs prototype rb lib/$file > sig/${file/rb/rbs}'

In my opinion, it would be much better if we had the above functionality by default (or maybe that's a feature—keeping all the signatures in the same file 🤔).

Also, copying comments from source files to signatures makes the latter less readable (especially if there are many comments, like in my case). Of course, we can add a bit more Unix magic to fix this...

Let's try runtime mode:

$ RUBYOPT="-Ilib" rbs prototype runtime -r rubanok Rubanok::Rule

class Rubanok::Rule
  public

  def activate_always: () -> untyped
  def activate_on: () -> untyped
  def applicable?: (untyped params) -> untyped
  def fields: () -> untyped
  def filter_with: () -> untyped
  def ignore_empty_values: () -> untyped
  def project: (untyped params) -> untyped
  def to_method_name: () -> untyped

  private

  def build_method_name: () -> untyped
  def empty?: (untyped val) -> untyped
  def fetch_value: (untyped params, untyped field) -> untyped
  def initialize: (untyped fields, ?activate_on: untyped, ?activate_always: untyped, ?ignore_empty_values: untyped, ?filter_with: untyped) -> untyped
end

In this mode, RBS uses Ruby introspection APIs (Class.methods, etc.) to generate the specified class or module signature.

Let's compare signatures for the Rubanok::Rule class generated with rb and runtime modes:

First, runtime generator does not recognize attr_reader (for instance, activate_on and activate_always).
Second, runtime generator sorts methods alphabetically while static generator preserves the original layout.
Finally, the first signature has a few types defined, while the latter has everything untyped.

So, why one may find runtime generator useful? I guess there are only one reason for that: dynamically generated methods. Like, for example, in Active Record.

Thus, both modes have their advantages and disadvantages and using them both would provide a better signature coverage. Unfortunately, there is no good way to diff/merge RBS files yet; you have to that manually. Another manual work is to replace untyped with the actual typing information.

But wait to make your hands dirty. There is one more player in this game–Type Profiler.

Type Profiler infers a program type signatures dynamically during the execution. It spies all the loaded classes and methods and collects the information about which types have been used as inputs and outputs, analyzes this data, and produces RBS definitions. Under the hood, it uses a custom Ruby interpreter (so, the code is not actually executed). You can find more in the official docs.

The main difference between TypeProf and RBS is that we need to create a sample script to be used as a profiling entry-point.

Let's write one:

# sig/rubanok_type_profile.rb
require "rubanok"

processor = Class.new(Rubanok::Processor) do
  map :q do |q:|
    raw
  end

  match :sort_by, :sort, activate_on: :sort_by do
    having "status", "asc" do
      raw
    end

    default do |sort_by:, sort: "asc"|
      raw
    end
  end
end

processor.project({q: "search", sort_by: "name"})
processor.call([], {q: "search", sort_by: "name"})

Now, let's run typeprof command:

$ typeprof -Ilib sig/rubanok_type_profile.rb --exclude-dir lib/rubanok/rails --exclude-dir lib/rubanok/rspec.rb

# Classes
module Rubanok
  VERSION : String

  class Rule
    UNDEFINED : Object
    @method_name : String
    attr_reader fields : untyped
    attr_reader activate_on : Array[untyped]
    attr_reader activate_always : false
    attr_reader ignore_empty_values : untyped
    attr_reader filter_with : nil
    def initialize : (untyped, ?activate_on: untyped, ?activate_always: false, ?ignore_empty_values: untyped, ?filter_with: nil) -> nil
    def project : (untyped) -> untyped
    def applicable? : (untyped) -> bool
    def to_method_name : -> String
    private
    def build_method_name : -> String
    def fetch_value : (untyped, untyped) -> Object?
    def empty? : (nil) -> false
  end

  # ...
end

Nice, now we have some types defined (though most of them are still untyped), we can see methods visibility and even instance variables (something we haven't seen before). The order of methods stayed the same as in the original file—that's good!

Unfortunately, despite being a runtime analyzer, TypeProf has not so good metaprogramming support. For example, the methods defined using iteration won't be recognized:

# a.rb
class A
  %w[a b].each.with_index { |str, i| define_method(str) { i } }
end

p A.new.a + A.new.b

$ typeprof a.rb

# Classes
class A
end

(We can handle this with rbs prototype runtime 😉)

So, even if you have an executable that provides 100% coverage of your APIs but uses metaprogramming, using just TypeProf is not enough to build a complete types scaffold for your program.

To sum up, all three different ways to generate initial signatures have their pros and cons, but combining their results could give a very good starting point in adding types to existing code. Hopefully, we'll be able to automate this in the future.

In Rubanok's case, I did the following:

Generating initial signatures using rbs prototype rb.
Ran typeprof and used its output to add missing instance variables and update some signatures.
Finally, ran rbs prototype runtime for main classes.

While I was writing this article, a PR with attr_reader self.foo support has been merged.

The latter one helped to find a bug in the signature generated at the first step:

 module Rubanok
-  attr_accessor ignore_empty_values: untyped
-  attr_accessor fail_when_no_matches: untyped
+  def self.fail_when_no_matches: () -> untyped
+  def self.fail_when_no_matches=: (untyped) -> untyped
+  def self.ignore_empty_values: () -> untyped
+  def self.ignore_empty_values=: (untyped) -> untyped
 end

Introducing Steep

So far, we've only discussed how to write and generate type signatures. That would be useless if we don't add a type checker to our dev stack.

As of today, the only type checker supporting RBS is Steep.

`steep init`

Let's add the steep gem to our dependencies and generate a configuration file:

steep init

That would generate a default Steepfile with some configuration. For Rubanok, I updated it like this:

# Steepfile
target :lib do
  # Load signatures from sig/ folder
  signature "sig"
  # Check only files from lib/ folder
  check "lib"

  # We don't want to type check Rails/RSpec related code
  # (because we don't have RBS files for it)
  ignore "lib/rubanok/rails/*.rb"
  ignore "lib/rubanok/railtie.rb"
  ignore "lib/rubanok/rspec.rb"

  # We use Set standard library; its signatures
  # come with RBS, but we need to load them explicitly
  library "set"
end

`steep stats`

Before drowning in a sea of types, let's think of how we can measure our signatures' efficiency. We can use steep stats to see how good (or bad?) our types coverage is:

$ bundle exec steep stats --log-level=fatal

Target,File,Status,Typed calls,Untyped calls,All calls,Typed %
lib,lib/rubanok/dsl/mapping.rb,success,7,2,11,63.64
lib,lib/rubanok/dsl/matching.rb,success,26,18,50,52.00
lib,lib/rubanok/processor.rb,success,34,8,49,69.39
lib,lib/rubanok/rule.rb,success,24,12,36,66.67
lib,lib/rubanok/version.rb,success,0,0,0,0
lib,lib/rubanok.rb,success,8,4,12,66.67

This command outputs surprisingly outputs CSV 😯. Let's add some Unix magic and make the output more readable:

$ bundle exec steep stats --log-level=fatal | awk -F',' '{ printf "%-28s %-9s %-12s %-14s %-10s\n", $2, $3, $4, $5, $7 }'
File                         Status    Typed calls  Untyped calls  Typed %   
lib/rubanok/dsl/mapping.rb   success   7            2              63.64     
lib/rubanok/dsl/matching.rb  success   26           18             52.00     
lib/rubanok/processor.rb     success   34           8              69.39     
lib/rubanok/rule.rb          success   24           12             66.67     
lib/rubanok/version.rb       success   0            0              0         
lib/rubanok.rb               success   8            4              66.67

Ideally, we would like to have everything typed. So, I opened my .rbs files and started replacing untyped with the actual types one by one.

It took me about a dozen minutes to get rid of untyped definitions (most of them). I'm not going to describe this process in detail; it was pretty straightforward except for the one thing I'd like to pay attention to.

Let's recall what Rubanok is. It provides a DSL to define data (usually, user input) transformers of a form (input, params) -> input. A typical use case is to customize an Active Record relation depending on request parameters:

class PagySearchyProcess < Rubanok::Processor
  map :page, :per_page, activate_always: true do |page: 1, per_page: 20|
   # raw is a user input
   raw.page(page).per(per_page)
  end

  map :q do |q:|
    raw.search(q)
  end
end

PagySearchyProcessor.call(Post.all, {q: "rbs"})
#=> Post.search("rbs").page(1).per(20)

PagySearchyProcessor.call(Post.all, {q: "rbs", page: 2})
#=> Post.search("rbs").page(2).per(20)

Thus, Rubanok deals with two external types: input (which could be anything) and params (which is a Hash with String or Symbol keys). Also, we have a notion of field internally: a params key used to activate a particular transformation. A lot of Rubanok's methods use these three entities, and to avoid duplication, I decided to use the type aliases feature of RBS:

module Rubanok
  # Transformation parameters
  type params = Hash[Symbol | String, untyped]
  type field = Symbol
  # Transformation target (we assume that input and output types are the same)
  type input = Object?

  class Processor
    def self.call: (params) -> input
                 | (input, params) -> input
    def self.fields_set: () -> Set[field]
    def self.project: (params) -> params

    def initialize: (input) -> void
    def call: (params) -> input
  end

  class Rule
    attr_reader fields: Array[field]

    def project: (params) -> params
    def applicable?: (params) -> bool
  end

  # ...
end

That allowed me to avoid duplication and indicate that they are not just Hashes, Strings, or whatever passing around, but params, fields and inputs.

Now, let's check our signatures!

Fighting with signatures, or make `steep check` happy

It's very unlikely that we wrote 100% correct signatures right away. I got ~30 errors:

$ bundle exec steep check --log-level=fatal

lib/rubanok/dsl/mapping.rb:24:8: MethodArityMismatch: method=map (def map(*fields, **options, &block))
lib/rubanok/dsl/mapping.rb:38:10: NoMethodError: type=(::Object & ::Rubanok::DSL::Mapping::ClassMethods), method=define_method (define_method(rule.to_method_name, &block))
lib/rubanok/dsl/mapping.rb:40:10: NoMethodError: type=(::Object & ::Rubanok::DSL::Mapping::ClassMethods), method=add_rule (add_rule rule)
lib/rubanok/dsl/matching.rb:25:10: MethodArityMismatch: method=initialize (def initialize(id, fields, values = [], **options, &block))
lib/rubanok/dsl/matching.rb:26:26: UnexpectedSplat: type= (**options)
lib/rubanok/dsl/matching.rb:29:12: IncompatibleAssignment:  ... 
lib/rubanok/dsl/matching.rb:30:32: NoMethodError: type=::Array[untyped], method=keys (@values.keys)
lib/rubanok/dsl/matching.rb:42:8: MethodArityMismatch: method=initialize (def initialize(*, **))
lib/rubanok/dsl/matching.rb:70:8: MethodArityMismatch: method=match (def match(*fields, **options, &block))
lib/rubanok/dsl/matching.rb:71:17: IncompatibleArguments: ...
lib/rubanok/dsl/matching.rb:73:10: BlockTypeMismatch: ...
lib/rubanok/dsl/matching.rb:75:10: NoMethodError: type=(::Object & ::Rubanok::DSL::Matching::ClassMethods), method=define_method (define_method(rule.to_method_name) do |params = {}|)
lib/rubanok/dsl/matching.rb:83:12: NoMethodError: type=(::Object & ::Rubanok::DSL::Matching::ClassMethods), method=define_method (define_method(clause.to_method_name, &clause.block))
lib/rubanok/dsl/matching.rb:86:10: NoMethodError: type=(::Object & ::Rubanok::DSL::Matching::ClassMethods), method=add_rule (add_rule rule)
lib/rubanok/dsl/matching.rb:96:15: NoMethodError: type=(::Object & ::Rubanok::DSL::Matching), method=raw (raw)
lib/rubanok/processor.rb:36:6: MethodArityMismatch: method=call (def call(*args))
lib/rubanok/processor.rb:56:13: NoMethodError: type=(::Class | nil), method=<= (superclass <= Processor)
lib/rubanok/processor.rb:57:12: NoMethodError: type=(::Class | nil), method=rules (superclass.rules)
lib/rubanok/processor.rb:67:13: NoMethodError: type=(::Class | nil), method=<= (superclass <= Processor)
lib/rubanok/processor.rb:68:12: NoMethodError: type=(::Class | nil), method=fields_set (superclass.fields_set)
lib/rubanok/processor.rb:78:21: ArgumentTypeMismatch: receiver=::Hash[::Symbol, untyped], expected=::Array[::Symbol], actual=::Set[::Rubanok::field] (*fields_set)
lib/rubanok/processor.rb:116:6: NoMethodError: type=::Rubanok::Processor, method=input= (self.input =)
lib/rubanok/processor.rb:134:6: NoMethodError: type=::Rubanok::Processor, method=input= (self.input = prepared_input)
lib/rubanok/rule.rb:11:6: IncompatibleAssignment: ...
lib/rubanok/rule.rb:20:8: UnexpectedJumpValue (next acc)
lib/rubanok/rule.rb:48:12: NoMethodError: type=(::Method | nil), method=call (filter_with.call(val))
lib/rubanok/rule.rb:57:8: MethodArityMismatch: method=empty? (def empty?)
lib/rubanok/rule.rb:63:8: MethodArityMismatch: method=empty? (def empty?)
lib/rubanok/rule.rb:69:4: MethodArityMismatch: method=empty? (def empty?(val))

Let's take a closer look at these errors and try to fix them.

1. Refinements always break things.

Let's start with the last three reported errors:

lib/rubanok/rule.rb:57:8: MethodArityMismatch: method=empty? (def empty?)
lib/rubanok/rule.rb:63:8: MethodArityMismatch: method=empty? (def empty?)
lib/rubanok/rule.rb:69:4: MethodArityMismatch: method=empty? (def empty?(val))

Why Steep detected three #empty? methods in the Rule class? It turned out that it considers an anonymous refinement body to be a part of the class body:

using(Module.new do
  refine NilClass do
    def empty?
      true
    end
  end

  refine Object do
    def empty?
      false
    end
  end
end)

def empty?(val)
  return false unless ignore_empty_values

  val.empty?
end

I submitted an issue and moved refinements to the top of the file to fix the errors.

2. Superclass don't cry 😢

Another interesting issue relates to superclass usage:

lib/rubanok/processor.rb:56:13: NoMethodError: type=(::Class | nil), method=<= (superclass <= Processor)
lib/rubanok/processor.rb:57:12: NoMethodError: type=(::Class | nil), method=rules (superclass.rules)
lib/rubanok/processor.rb:67:13: NoMethodError: type=(::Class | nil), method=<= (superclass <= Processor)

The corresponding source code:

@rules =
  if superclass <= Processor
    superclass.rules.dup
  else
    []
  end

It's a very common pattern to inherit class properties. Why doesn't it work? First, the superclass signature says the result is either Class or nil (though it could be nil only for the BaseObject class, as far as I know). Thus, we cannot use <= right away (because it's not defined on NilClass.

Even if we unwrap superclass, the problem with .rules would still be there—Steep's flow sensitivity analysis currently doesn't recognize the <= operator. So, I decided to hack the system and explicitly define the .superclass signature for the Processor class:

# processor.rbs
class Processor
  def self.superclass: () -> singleton(Processor)
  # ...
end

This way, my code stays the same; only the types suffer 😈.

3. Explicit over implicit: handling splats.

So far, we've seen pretty much the same problems as I had with Sorbet. Let's take a look at something new.

Consider this code snippet:

def project(params)
  params = params.transform_keys(&:to_sym)
  # params is a Hash, fields_set is a Set
  params.slice(*fields_set)
end

It produces the following type error:

lib/rubanok/processor.rb:78:21: ArgumentTypeMismatch: receiver=::Hash[::Symbol, untyped], expected=::Array[::Symbol], actual=::Set[::Rubanok::field]

The Hash#slice method expects an Array, but we pass a Set. However, we also use a splat (*) operator, which implicitly tries to convert an object to an array—seems legit, right? Unfortunately, Steep is not so smart yet: we have to add an explicit #to_a call.

4. Explicit over implicit, pt. 2: forwarding arguments.

I used the following pattern in a few places:

def match(*fields, **options, &block)
  rule = Rule.new(fields, **options)

  # ...
end

A DSL method accepts some options as keyword arguments and then pass them to the Rule class initializer. The possible options are strictly defined and enforced in the Rule#initialize, but we would like to avoid declaring them explicitly just to forward down. Unfortunately, that's only possible if we declare **options as untyped—that would make signatures kinda useless.

So, we have to become more explicit once again:

-        def map(*fields, **options, &block)
-          filter = options[:filter_with]
-          rule = Rule.new(fields, **options)

+        def map(*fields, activate_on: fields, activate_always: false, ignore_empty_values: Rubanok.ignore_empty_values, filter_with: nil, &block)
+          filter = filter_with
+          rule = Rule.new(fields, activate_on: activate_on, activate_always: activate_always, ignore_empty_values: ignore_empty_values, filter_with: filter_with)

# and more...

I guess it's time to add Ruby Next and use shorthand Hash notation 🙂

5. Variadic arguments: annotations to the rescue!

In the recent Rubanok release, I added an ability to skip input for transformations and only use params as the only #call method argument. That led to the following code:

def call(*args)
  input, params =
  if args.size == 1
    [nil, args.first]
  else
    args
  end

  new(input).call(params)
end

As in the previous case, we needed to make our signature more explicit and specify the actual arguments instead of the *args:

# This is our signature
# (Note that we can define multiple signatures for a method)
def self.call: (input, params) -> input
             | (params) -> input

# And this is our code (first attempt)
UNDEFINED = Object.new

def call(input, params = UNDEFINED)
  input, params = nil, input if params == UNDEFINED

  raise ArgumentError, "Params could not be nil" if params.nil?

  new(input).call(params)
end

This refactoring doesn't pass the type check:

$ bundle exec steep lib/rubanok/processor.rb

lib/rubanok/processor.rb:43:24: ArgumentTypeMismatch: receiver=::Rubanok::Processor, expected=::Rubanok::params, actual=(::Rubanok::input | ::Rubanok::params | ::Object) (params)

So, according to Steep, param could be pretty match anything :( We need to help Steep to make the right decision. I couldn't find a way to do that via RBS, so my last resort was to use annotations.

Yes, even though RBS itself is designed not to pollute your source code, Steep allows you to do that. And in some cases, that's the necessary evil.

I came up with the following:

def call(input, params = UNDEFINED)
  input, params = nil, input if params == UNDEFINED

  raise ArgumentError, "Params could not be nil" if params.nil?

  # @type var params: untyped
  new(input).call(params)
end

We declare params as untyped to silence the error. The #call method signature guarantees that the params variable satisfies the params type requirements, so we should be safe here.

6. Deal with metaprogramming: interfaces.

Since Rubanok provides a DSL, it heavily uses metaprogramming.
For example, we use #define_method to dynamically generate transformation methods:

def map(*fields, activate_on: fields, activate_always: false, ignore_empty_values: Rubanok.ignore_empty_values, filter_with: nil, &block)
  # ...
  rule = Rule.new(fields, activate_on: activate_on, activate_always: activate_always, ignore_empty_values: ignore_empty_values, filter_with: filter_with)

  define_method(rule.to_method_name, &block)

  add_rule rule
end

And that's the error we see when running steep check:

lib/rubanok/dsl/mapping.rb:38:10: NoMethodError: type=(::Object & ::Rubanok::DSL::Mapping::ClassMethods), method=define_method (define_method(rule.to_method_name, &block))
lib/rubanok/dsl/mapping.rb:40:10: NoMethodError: type=(::Object & ::Module & ::Rubanok::DSL::Mapping::ClassMethods), method=add_rule (add_rule rule)

Hmm, looks like our type checker doesn't know that we're calling the .map method in the context of the Processor class (we call Processor.extend DSL::Mapping).

RBS has a concept of a self type for module: a self type adds requirements to the classes/modules, which include/prepend/extend this module. For example, we can state that we only allow using Mapping::ClassMethods to extend modules (and not objects, for example):

# Module here is a self type
module ClassMethods : Module
  # ...
end

That fixes NoMethodError for #define_method, but we still have it for #add_rule—this is a Processor self method. How can we add this restriction using module self types? It's not allowed to use singleton(SomeClass) as a self type; only classes and interfaces are allowed. Yes, RBS has interfaces! Let's give them a try!

We only use the #add_rule method in the modules, so we can define an interface as follows:

interface _RulesAdding
  def add_rule: (Rule rule) -> void
end

# Then we can use this interface in the Processor class itself
class Processor
  extend _RulesAdding

  # ...
end

# And in our modules
module Mapping
  module ClassMethods : Module, _RulesAdding
    # ...
  end
end

7. Making Steep happy.

Other problems I faced with Steep which I converted into issues:

Method visibility false negatives

Inability to declare block return values in some cases

I added a few more changes to the signatures and the source code to finally make a steep check pass. The journey was a bit longer than I expected, but in the end, I'm pretty happy with the result—I will continue using RBS and Steep.

Here is the final stats for Rubanok:

File                         Status    Typed calls  Untyped calls  Typed %   
lib/rubanok/dsl/mapping.rb   success   11           0              100.00    
lib/rubanok/dsl/matching.rb  success   54           2              94.74     
lib/rubanok/processor.rb     success   52           2              96.30     
lib/rubanok/rule.rb          success   31           2              93.94     
lib/rubanok/version.rb       success   0            0              0         
lib/rubanok.rb               success   12           0              100.00

Runtime type checking with RBS

Although RBS doesn't provide static type checking capabilities, it comes with runtime testing utils. By loading a specific file (rbs/test/setup), you can ask RBS to watch the execution and check that method calls inputs and outputs satisfy signatures.

Under the hood, TracePoint API is used along with the alias method chain trick to hijack observed methods. Thus, it's meant for use in tests, not in production.

Let's try to run our RSpec tests with runtime checking enabled:

$ RBS_TEST_TARGET='Rubanok::*' RUBYOPT='-rrbs/test/setup' bundle exec rspec --fail-fast

I, [2020-12-07T21:07:57.221200 #285]  INFO -- : Setting up hooks for ::Rubanok
I, [2020-12-07T21:07:57.221302 #285]  INFO -- rbs: Installing runtime type checker in Rubanok...
...

Failures:

  1) Rails controllers integration PostsApiController#planish implicit rubanok with matching
     Failure/Error: prepare! unless prepared?

     RBS::Test::Tester::TypeError:
       TypeError: [Rubanok::Processor#prepared?] ReturnTypeError: expected `bool` but returns `nil`

Oh, we forgot to initialize the @prepared instance variable with the boolean value! Nice!

When I tried to use RBS runtime tests for the first time, I encountered a few severe problems. Many thanks to Soutaro Matsumoto for fixing all of them faster than I finished working on this article!

I found a couple of more issues by using rbs/test/setup, including the one I wasn't able to resolve:

Failure/Error: super(fields, activate_on: activate_on, activate_always: activate_always)

RBS::Test::Tester::TypeError:
  TypeError: [Rubanok::Rule#initialize] UnexpectedBlockError: unexpected block is given for `(::Array[::Rubanok::field] fields, ?filter_with: ::Method?, ?ignore_empty_values: bool, ?activate_always: bool, ?activate_on: ::Rubanok::field | ::Array[::Rubanok::field]) -> void`

And here is the reason:

class Clause < Rubanok::Rule
  def initialize(id, fields, values, **options, &block)
    # The block is passed to super implicitly,
    # but is not acceptable by Rule#initialize
    super(fields, **options)
  end
end

I tried to use &nil to disable block propagation, but that broke steep check 😞. I submitted an issue and excluded Rule#initialize from the runtime checking for now using a special comment in the .rbs file:

# rule.rbs
class Rule
  # ...
  %a{rbs:test:skip} def initialize: (
    Array[field] fields,
    ?activate_on: field | Array[field],
    ?activate_always: bool,
    ?ignore_empty_values: bool,
    ?filter_with: Method?
  ) -> void
end

Bonus: Steep meets Rake

I usually run be rake pretty often during development to make sure that everything is correct. The default task usually includes RuboCop and tests.

Let's add Steep to the party:

# Rakefile

# other tasks

task :steep do
  # Steep doesn't provide Rake integration yet,
  # but can do that ourselves 
  require "steep"
  require "steep/cli"

  Steep::CLI.new(argv: ["check"], stdout: $stdout, stderr: $stderr, stdin: $stdin).run
end

namespace :steep do
  # Let's add a user-friendly shortcut
  task :stats do
    exec %q(bundle exec steep stats --log-level=fatal | awk -F',' '{ printf "%-28s %-9s %-12s %-14s %-10s\n", $2, $3, $4, $5, $7 }')
  end
end

# Run steep before everything else to fail-fast
task default: %w[steep rubocop rubocop:md spec]

Bonus 2: Type Checking meets GitHub Actions

As the final step, I configure GitHub Actions to run both static and runtime type checks:

# lint.yml
jobs:
  steep:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: 2.7
    - name: Run Steep check
      run: |
        gem install steep
        steep check

# rspec.yml
jobs:
  rspec:
    # ...
    steps:
      # ...
    - name: Run RSpec with RBS
      if: matrix.ruby == '2.7'
      run: |
        gem install rbs
        RBS_TEST_TARGET="Rubanok::*" RUBYOPT="-rrbs/test/setup" bundle exec rspec --force-color
    - name: Run RSpec without RBS
      if: matrix.ruby != '2.7'
      run: |
        bundle exec rspec --force-color

Although there are still enough rough edges, I enjoyed using RBS/Steep a bit more than "eating" Sorbet (mostly because I'm not a big fan of type annotations in the source code). I will continue adopting Ruby 3 types in my OSS projects and reporting as many issues to RBS/Steep as possible 🙂.

P.S. You can find the source code in this PR.

RuboCoping with legacy

Vladimir Dementyev — Tue, 24 Mar 2020 21:38:35 +0000

Originally posted in Martian Chronicles.

You will hardly find a Ruby developer who hasn't heard about RuboCop, the Ruby linter and formatter. And still, it is not that hard to find a project where code style is not enforced. Usually, these are large, mature codebases, often successful ones. Fixing linting and formatting can be a challenge if it wasn't set up correctly from the get-go. So, your RuboCop sees red! Here's how to fix it.

In this post, I will show you how we at Evil Martians touch up codebases of our customers in 2020: from quick and dirty hacks to proper Standard-enforced style guides, and our own ~~patented~~ way to use Standard and RuboCop configs together.

Style matters

Let's pretend I have to convince you to follow code style guidelines (I know, I know I don't have to!)

Here are the arguments I would use:

Developers understand each other much better when they ~~speak~~ write the same language.
Onboarding new engineers becomes much easier when the code style is standardized.
Linters help to detect and squash bugs in time.
No more "single vs. double quotes" holy wars (double FTW)!

That was all the theory for today. Time for practice!

TODO or not TODO

So, you have joined a project with no style guide or with a .rubocop.yml that was added years ago. You run RuboCop, and you see something like:

$ bundle exec rubocop

3306 files inspected, 12418 offenses detected

Flocks of noble ~~knights~~ developers tried to ~~slay the beast~~ fix the offenses but gave up. But that doesn't stop you—you know the magic spell:

$ bundle exec rubocop --auto-gen-config
Added inheritance from `.rubocop_todo.yml` in `.rubocop.yml`.
Created .rubocop_todo.yml.

$ bundle exec rubocop
3306 files inspected, no offenses detected

That was simple! Toss the coin to your...

Let's take a closer look at what --auto-gen-config flag does:

First, it collects all the offenses and their counts;
then, it generates a .rubocop_todo.yml where all the current offenses are ignored;
and finally, it makes .rubocop.yml inherit from .rubocop_todo.yml.

That is the way to set the status quo and only enforce style checks for new code. Sounds smart, right? Not exactly.

The way .rubocop_todo.yml handles "ignores" depends on the cop types and the total number of current offenses:

For metrics cops (such as Layout/LineLength), the limit (Max) is set to the maximum value for the current codebase.
All cops could be disabled if the total number of offenses hits the threshold (only 15 by default).

So, you end up with anything goes situation, and that defeats the purpose.

This article covers this problem in more detail.

What does it mean for a typical legacy codebase? Most of the new code would be ignored by RuboCop, too. We made the tool happy, but are we happy with it?

Hopefully, there is a way to generate a better TODO config by adding more options to the command:

bundle exec rubocop –-auto-gen-config \
  --auto-gen-only-exclude \
  --exclude-limit=10000

Where --auto-gen-only-exclude force-excludes metrics cops instead of changing their Max value, and --exclude-limit sets the threshold for the exclusion (set to some large enough number to avoid disabling cops completely).

Now your .rubocop_todo.yml won't affect your new files or entirely new offenses in the old ones.

RuboCop doesn't only help with style—it also saves you from common mistakes that can break your code in production. What if you had some bugs and ignored them in your TODO config? What are the cops that should never be ignored? Let me introduce the RuboCop strict configuration pattern.

You shall not pass: introducing `.rubocop_strict.yml`

There are a handful of cops that must be enabled for all the files independently of the .rubocop_todo.yml. For example:

Lint/Debugger—don't leave debugging calls (e.g., binding.pry).
RSpec/Focus (from rubocop-rspec)—don't forget to clear focused tests (to make sure CI runs the whole test suite).

We put such cops into a .rubocop_strict.yml configuration file like this:

inherit_from:
  - .rubocop_todo.yml

Lint/Debugger: # don't leave binding.pry
  Enabled: true
  Exclude: []

RSpec/Focus: # run ALL tests on CI
  Enabled: true
  Exclude: []

Rails/Output: # Don't leave puts-debugging
  Enabled: true
  Exclude: []

Rails/FindEach: # each could severely affect the performance, use find_each
  Enabled: true
  Exclude: []

Rails/UniqBeforePluck: # uniq.pluck and not pluck.uniq
  Enabled: true
  Exclude: []

Then, we replace the TODO config with the Strict config in our base .rubocop.yml configuration file:

 inherit_from:
-  - .rubocop_todo.yml
+  - .rubocop_strict.yml

The Exclude: [] is crucial here: even if our .rubocop_todo.yml contained exclusions for strict cops, we nullify them here, thus, re-activating these cops for all the files.

One Standard to rule them all

One of the biggest problems in adopting a code style is to convince everyone on the team to always use double-quotes for strings, or to add trailing commas to multiline arrays, or ro <choose-your-own-controversal-style-rule>? We are all well familiar with bikeshedding.

RuboCop provides a default configuration based on the Ruby Style Guide. And you know what? It's hard to find a project which follows all of the default rules, there are always reconfigured or disabled cops in the .rubocop.yml.

That's okay. RuboCop's default configuration is not a golden standard; it was never meant to be the one style to fit them all.

Should Ruby community have the only style at all? It seems that yes, we need it.

I think the main reason for that is the popularity of auto-formatters in other programming languages: JavaScript, Go, Rust, Elixir. Auto-formatters are usually very strict and allow almost none or zero configuration. And developers got used to that! People like writing code without worrying about indentation, brackets, and spaces; robots would sort it all out!

Checkout out the lightning talk and let Justin Searls convince you to switch to Standard.

Thankfully, Ruby's ecosystem has got you covered: there is a project called Standard, which claims to be the one and only Ruby style guide.

From the technical point of view, Standard is a wrapper over RuboCop with its custom configuration and CLI (standard).

Unfortunately, Standard lacks some RuboCop features that are essential at the early stages of a style guide's adoption: it is not possible to use .rubocop_todo.yml or any other local configuration. It also doesn't support RuboCop plugins or custom cops.

But we can still use Standard as a style guide while continuing to use RuboCop as a linter and formatter!

For that, we can use RuboCop's inherit_gem directive:

# .rubocop.yml

# We want Exclude directives from different
# config files to get merged, not overwritten
inherit_mode:
  merge:
    - Exclude

require:
  # Performance cops are bundled with Standard
  - rubocop-performance
  # Standard's config uses this custom cop,
  # so it must be loaded
  - standard/cop/semantic_blocks

inherit_gem:
  standard: config/base.yml

inherit_from:
  - .rubocop_strict.yml

# Sometimes we enable metrics cops
# (which are disabled in Standard by default)
#
# Metrics:
#   Enabled: true

That is the configuration I use in most of my OSS and commercial projects. I can't say I agree with all the rules, but I definitely like it more than the RuboCop's default. That is a tiny trade-off if you think about the benefit of not arguing over the style anymore.

Don't forget to add standard to your Gemfile and freeze its minor version to avoid unexpected failures during upgrades:

gem "standard", "~> 0.2.0", require: false

Although the approach above allows you to tinker with the Standard configuration, I would not recommend doing that. Use this flexibility to extend the default behavior, not change it!

Beyond the Standard

RuboCop has a lot of plugins distributed as separate gems: rubocop-rspec, rubocop-performance, rubocop-rails, rubocop-md, to name a few.

Standard only includes the rubocop-performance plugin. We usually add rubocop-rails and rubocop-rspec to our configuration.

For each plugin, we keep a separate YAML file: .rubocop_rails.yml, .rubocop_rspec.yml, etc.

Inside the base config we add these files to inherit_from:

inherit_from:
  - .rubocop_rails.yml
  - .rubocop_rspec.yml
  - .rubocop_strict.yml

Our .rubocop_rails.yml is based on the configuration that existed in Standard before they dropped Rails support.

There is no standard RSpec configuration, so we had to figure out our own: .rubocop_rspec.yml.

We also usually enable a select few custom cops, for example, Lint/Env.

In the end, our typical RuboCop configuration for Rails projects looks like this👇

# .rubocop.yml
inherit_mode:
  merge:
    - Exclude

require:
  - rubocop-performance
  - standard/cop/semantic_blocks
  - ./lib/cops/lint/env.rb

inherit_gem:
  standard: config/base.yml

inherit_from:
  - .rubocop_rails.yml
  - .rubocop_rspec.yml
  - .rubocop_strict.yml

Lint/Env:
  Enabled: true
  Include:
    - '**/*.rb'
  Exclude:
    - '**/config/environments/**/*'
    - '**/config/application.rb'
    - '**/config/environment.rb'
    - '**/config/puma.rb'
    - '**/config/boot.rb'
    - '**/spec/*_helper.rb'
    - '**/spec/**/support/**/*'
    - 'lib/generators/**/*'

Feel free to use it as an inspiration for your projects that could use some RuboCop's tough love.

RuboCop plays a vital role in the Ruby world and will stay TOP-1 for linting and formatting code for quite a long time (though competing formatters are evolving, for example, rubyfmt and prettier-ruby). Don't ignore RuboCop; write code in style 😎

Read more dev articles on https://evilmartians.com/chronicles!

Pulling the trigger: How to update counter caches in your Rails app without Active Record callbacks

Dmitry Tsepelev — Tue, 26 Nov 2019 10:13:09 +0000

In this article, we experiment with triggers as a tool for keeping aggregated data consistent when using Active Record and your favorite SQL database. Instead of using sophisticated tools such as ElasticSearch for filtering and searching, we will demonstrate a simple approach that achieves the same result with some out-of-the-box database features. As a bonus, learn how to avoid nasty race conditions!

You can find all the examples in the gist.

Sometimes you need to sort and filter records in the database by some aggregated values. For instance, you might be building a paginated list of users in an admin panel, and you want to implement filtering by the number of orders and the total amount that users have spent on them. There are several tools like ElasticSearch, which are good at filtering by aggregates, but setting up a massive search engine and all required infrastructure to process a couple of columns sounds like an overkill. Let's find a more straightforward way!

Trigger finger

Imagine the following data model:

ActiveRecord::Schema.define do
  create_table "orders", force: :cascade do |t|
    t.bigint "user_id", null: false
    t.decimal "amount"
    t.datetime "created_at", precision: 6, null: false
    t.datetime "updated_at", precision: 6, null: false
    t.index ["user_id"], name: "index_orders_on_user_id"
  end

  create_table "users", force: :cascade do |t|
    t.datetime "created_at", precision: 6, null: false
    t.datetime "updated_at", precision: 6, null: false
  end

  add_foreign_key "orders", "users"
end

class User < ActiveRecord::Base
  has_many :orders
end

class Order < ActiveRecord::Base
  belongs_to :user
end

Let's see how we can filter and paginate users by their order total. We can easily achieve our goal with the vanilla SQL statement, but we will immediately run into performance issues. To demonstrate, let's fill the database with 10,000 users and 100,000 orders and use explain (you can find a single file implementation of this example in this gist):

User.insert_all(10_000.times.map { { created_at: Time.now, updated_at: Time.now } })

Order.insert_all(
  10_000.times.map do
    {
      user_id: rand(1...1000),
      amount: rand(1000) / 10.0,
      created_at: Time.now,
      updated_at: Time.now
    }
  end
)

ActiveRecord::Base.connection.execute <<~SQL
  EXPLAIN ANALYZE SELECT users.id, SUM(orders.amount), COUNT(orders.id)
  FROM users JOIN orders ON orders.user_id = users.id
  GROUP BY users.id
  HAVING SUM(orders.amount) > 100 AND COUNT(orders.id) > 1
  ORDER BY SUM(orders.amount)
  LIMIT 50
SQL

This is the result you might see:

Limit  (cost=3206.16..3206.29 rows=50 width=48) (actual time=59.737..59.746 rows=50 loops=1)
  ->  Sort  (cost=3206.16..3208.95 rows=1116 width=48) (actual time=59.736..59.739 rows=50 loops=1)
        Sort Key: (sum(orders.amount))
        Sort Method: top-N heapsort  Memory: 31kB
        ->  HashAggregate  (cost=2968.13..3169.09 rows=1116 width=48) (actual time=59.103..59.452 rows=1000 loops=1)
              Group Key: users.id
              Filter: ((sum(orders.amount) > '100'::numeric) AND (count(orders.id) > 1))
              ->  Hash Join  (cost=290.08..2050.73 rows=73392 width=48) (actual time=2.793..37.022 rows=100000 loops=1)
                    Hash Cond: (orders.user_id = users.id)
                    ->  Seq Scan on orders  (cost=0.00..1567.92 rows=73392 width=48) (actual time=0.011..11.650 rows=100000 loops=1)
                    ->  Hash  (cost=164.48..164.48 rows=10048 width=8) (actual time=2.760..2.760 rows=10000 loops=1)
                          Buckets: 16384  Batches: 1  Memory Usage: 519kB
                          ->  Seq Scan on users  (cost=0.00..164.48 rows=10048 width=8) (actual time=0.006..1.220 rows=10000 loops=1)
Planning Time: 0.237 ms
Execution Time: 64.151 ms

With a bigger database, it will take even more time! We need to find a better solution, the one that scales. Let's denormalize our database and store orders_amount in the separate user_stats table:

class CreateUserStats < ActiveRecord::Migration[6.0]
  def change
    create_table :user_stats do |t|
      t.integer :user_id, null: false, foreign_key: true
      t.decimal :orders_amount
      t.integer :orders_count

      t.index :user_id, unique: true
    end
  end
end

Now we should decide how to keep orders_count and orders_amount in sync. ActiveRecord callbacks do not look like a proper place to handle such operations, because we want to have our stats updated even when data is changed with a plain SQL (e.g., in the migration). There is a built-in counter_cache option for the belongs_to association, but it cannot help us with orders_amount. Triggers to the rescue!

A trigger is a function that is automatically invoked when INSERT, UPDATE, or DELETE is performed on the table.

To work with triggers from our Rails app, we can use gems like hair_trigger, fx, or even write them by hand. In this example, we use hair_trigger, which can generate migrations for trigger updates using only the latest version of the SQL procedure.

Heads up! There is a known hair_trigger issue with Rails 6 and Zeitwerk, if you face it–feel free to use my fork for now. Don't forget to switch back when the fix is out!

Let's add our trigger to the Order model. We want to perform the UPSERT: if there is no row with the matching user_id in the user_stats table–we add a new row, otherwise–update the existing one (make sure there is a unique constraint on the user_id column):

class Order < ActiveRecord::Base
  belongs_to :user

  trigger.after(:insert) do
    <<~SQL
      INSERT INTO user_stats (user_id, orders_amount, orders_count)
      SELECT
        NEW.user_id as user_id,
        SUM(orders.amount) as orders_amount,
        COUNT(orders.id) as orders_count
      FROM orders WHERE orders.user_id = NEW.user_id
      ON CONFLICT (user_id) DO UPDATE
      SET
        orders_amount = EXCLUDED.orders_amount,
        orders_count = EXCLUDED.orders_count;
    SQL
  end
end

Now we should generate the migration with rake db:generate_trigger_migration, run migrations with rails db:migrate, and run the application.

Off to the races

It might seem to be working, but what if we try to insert multiple orders in parallel? (you can run the following code as a rake task or check my implementation here)

user = User.create

threads = []

4.times do
  threads << Thread.new(user.id) do |user_id|
    user = User.find(user_id)
    user.orders.create(amount: rand(1000) / 10.0)
  end
end

threads.each(&:join)

inconsistent_stats = UserStat.joins(user: :orders)
                             .where(user_id: user.id)
                             .having("user_stats.orders_amount <> SUM(orders.amount)")
                             .group("user_stats.id")

if inconsistent_stats.any?
  calculated_amount = UserStat.find_by(user: user).orders_amount
  real_amount = Order.where(user: user).sum(:amount).to_f

  puts
  puts "Race condition detected:"
  puts "calculated amount: #{calculated_amount}"
  puts "real amount: #{real_amount}."
else
  puts
  puts "Data is consistent."
end

There is a huge chance that there will be a race condition, but why? The problem is that the trigger runs inside the current transaction, and the default isolation level is READ COMMITTED, which cannot handle race conditions.

PostgreSQL supports four levels of transaction isolation–READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ and SERIALIZABLE

The obvious solution is to use a stricter SERIALIZABLE isolation level, but, unfortunately, an isolation level cannot be changed inside a running transaction. Creating a new explicit transaction every time we work with orders does not sound right either, so let's try another approach for making sure our triggers are always executed in sequence–advisory locks.

The only thing we need to change is to add lock PERFORM pg_advisory_xact_lock(NEW.user_id); at the beginning of our procedure code:

class Order < ActiveRecord::Base
  belongs_to :user

  trigger.after(:insert) do
    <<~SQL
      PERFORM pg_advisory_xact_lock(NEW.user_id);

      INSERT INTO user_stats (user_id, orders_amount, orders_count)
      SELECT
        NEW.user_id as user_id,
        SUM(orders.amount) as orders_amount,
        COUNT(orders.id) as orders_count
      FROM orders WHERE orders.user_id = NEW.user_id
      ON CONFLICT (user_id) DO UPDATE
      SET
        orders_amount = EXCLUDED.orders_amount,
        orders_count = EXCLUDED.orders_count;
    SQL
  end
end

It's way faster! The updated version of the code is here, if you run it, you'll see that race condition is gone, and the app can handle parallel requests. Let's add the index to the orders_amount column in the user_stats table, change the query, and compare the performance:

EXPLAIN ANALYZE SELECT user_id, orders_amount, orders_count
FROM user_stats
WHERE orders_amount > 100 AND orders_count > 1
ORDER BY orders_amount
LIMIT 50

Limit  (cost=0.29..22.99 rows=50 width=40) (actual time=0.059..11.241 rows=50 loops=1)
  ->  Index Scan using index_user_stats_on_orders_amount on user_stats  (cost=0.29..3438.69 rows=7573 width=40) (actual time=0.058..11.2 rows=50 loops=1)
        Index Cond: (orders_amount > '100'::numeric)
        Filter: (orders_count > 1)
Planning Time: 0.105 ms
Execution Time: 11.272 ms

Lock-free alternative

There is a way (suggested by Sergey Ponomarev) to achieve the same result without locks and make it work faster—use deltas (you can find the full implementation here):

class Order < ActiveRecord::Base
  belongs_to :user

  trigger.after(:insert) do
    <<~SQL
      INSERT INTO user_stats (user_id, orders_amount, orders_count)
      SELECT
        NEW.user_id as user_id,
        NEW.amount as orders_amount,
        1 as orders_count
      ON CONFLICT (user_id) DO UPDATE
      SET
        orders_amount = user_stats.orders_amount + EXCLUDED.orders_amount,
        orders_count = user_stats.orders_count + EXCLUDED.orders_count;
    SQL
  end
end

The trick here is not to use any subqueries, so race conditions would not be possible. As a bonus, you'll get better performance when inserting new records. This approach might come handy for simple cases like the one described in this article, but when you are dealing with more complex logic, you might want to resort to locks (imagine that orders have statuses, we need to cache counts of orders in each status and orders can be updated).

Loop instead of UPSERT

In previous examples, we use UPSERT, which was introduced in PostgreSQL 9.5, but what if we use the older version? Let's review how the trigger works again: it tries to insert a new row into the user_stats table and, if a conflict happens, it updates the existing row. In the real-world application there will be conflicts most of the time (to be precise–insert happens only once for each user). We can use this fact and rewrite our trigger in the following way (the example of a loop inside the trigger is here):

class Order < ActiveRecord::Base
  belongs_to :user

  trigger.after(:insert) do
    <<~SQL
      <<insert_update>>
      LOOP
        UPDATE user_stats
        SET orders_count = orders_count + 1,
            orders_amount = orders_amount + NEW.amount
        WHERE user_id = NEW.user_id;

        EXIT insert_update WHEN FOUND;

        BEGIN
          INSERT INTO user_stats (
            user_id, orders_amount, orders_count
          ) VALUES (
            NEW.user_id, 1, NEW.amount
          );

          EXIT insert_update;
        EXCEPTION
          WHEN UNIQUE_VIOLATION THEN
            -- do nothing
        END;
      END LOOP insert_update;
    SQL
  end
end

In this case, we have inverted our logic: trigger tries to update the existing row, and if it misses—the new row gets inserted.

Working with aggregated data is hard: when you have a lot of counter (and other kinds of) caches, it makes sense to use a special tool for that. However, for simple cases, we can stay with good old database triggers: when configured properly, they are quite performant!

Persisted queries in GraphQL: Slim down Apollo requests to your Ruby application

Dmitry Tsepelev — Thu, 07 Nov 2019 13:24:52 +0000

Learn how to reduce the size of network requests from the Apollo client in the front-end to the GraphQL Ruby back-end with the help of persisted queries. In this article, we will show how these queries work and set them up both on a client and a server with the help of our graphql-ruby-persisted_queries Ruby gem.

One of the benefits of using GraphQL is its flexibility: back-end describes the data model using types, so front-end can only get the data it needs. The problem is that the amount of data required to render a page in a real-world application can be significant, and the query to fetch this data will get out of hand pretty quick. What could help here?

Read how Relay deals with persisted queries here.

First of all, the amount and the variety of often-used queries in your application are limited: usually, front-end knows precisely what it needs from the back-end for every particular view. Popular GraphQL frameworks like Apollo and Relay use that fact to persist queries in the back-end so that a front-end can send just a unique ID over the wire, instead of the full query in all its verboseness. This article will focus on the Apollo implementation on persisted queries.

If you have the Pro Version of graphql-ruby–you already have built-in support for persisted queries.

Most of Ruby applications that implement GraphQL server use a gem called graphql-ruby, in this article we are going to find out how to make it work with persisted query IDs coming from the Apollo Client. By default, graphql-ruby uses POST requests, as query strings tend to become very long, and it makes it hard to configure HTTP caching. If we switch to persisted queries, we will be able to turn on GET requests too and unleash the power of HTTP caching!

The idea behind persisting queries is pretty straightforward–the back-end should look for a special query parameter, containing the unique ID of the query, find the query in the store (we'll talk about it later, imagine that it's just a key-value store) and use it to prepare the response.

Relay has a compiler, and it knows all the queries in the app, so it can put them to a dedicated text file along with their md5 hashes when --persist-output option is provided. You can save these queries to your store, and you're done: back-end is ready to consume IDs instead of full queries.

Apollo has no compiler, and there is no built-in way to get the list of queries. In this case, queries should be saved to a store at runtime, and there is a special apollo-link-persisted-queries library that enables this feature. First, change your frontend configuration like so:

import { createPersistedQueryLink } from "apollo-link-persisted-queries";
import { createHttpLink } from "apollo-link-http";
import { InMemoryCache } from "apollo-cache-inmemory";
import ApolloClient from "apollo-client";

const link = createPersistedQueryLink().concat(createHttpLink({ uri: "/graphql" }));
const client = new ApolloClient({
  cache: new InMemoryCache(),
  link: link,
});

Now, let's talk about the back-end. With the configured link, client will make an attempt to send the unique ID (sha256 is used in this case) in the extension param, e.g.:

{
  extensions: {
    persistedQuery: {
      version: 1,
      sha256Hash: "688787d8ff144c502c7f5cffaafe2cc588d86079f9de88304c26b0cb99ce91c6"
    }
  }
}

When server finds the ID in the store—it serves the query if it were sent in full, otherwise–server returns { errors: [{ message: "PersistedQueryNotFound" }] } in the response payload. In this case client will send the full query along with unique ID and back-end will persist it to the store.

Now we need to implement our back-end functionality in the GraphqlController:

class GraphqlController < ApplicationController
  PersistedQueryNotFound = Class.new(StandardError)

  def execute
    query_str = persisted_query || params[:query]

    result = GraphqlSchema.execute(
      query_str,
      variables: ensure_hash(params[:variables]),
      context: {},
      operation_name: params[:operationName]
    )

    render json: result
  rescue PersistedQueryNotFound
    render json: { errors: [{ message: "PersistedQueryNotFound" }] }
  end

  private

  def persisted_query
    return if params[:extensions].nil?

    hash = ensure_hash(params[:extensions]).dig("persistedQuery", "sha256Hash")

    return if hash.nil?

    if params[:query]
      store.save_query(hash, params[:query])
      return
    end

    query_str = store.fetch_query(hash)

    raise PersistedQueryNotFound if query_str.nil?

    query_str
  end

  def store
    MemoryStore.instance
  end

  def ensure_hash(ambiguous_param)
    # ...
  end
end

Now we need to implement the MemoryStore (as we know, we need a very simple key-value storage with read and write operations), and that's it, minimalistic back-end support of persisted queries is ready:

class MemoryStore
  def self.instance
    @instance ||= new
  end

  def initialize
    @storage = {}
  end

  def fetch_query(hash)
    @storage[hash]
  end

  def save_query(hash, query)
    @storage[hash] = query
  end
end

And that is it! If you configured your client properly and changed your GraphqlController, it should work! So you don't have to copy all the boilerplate above between projects, I released a little gem called graphql-ruby-persisted_queries, which implements all the features we discussed earlier, and more:

Redis storage;
hash verification;
hash function configuration;
Relay support is on its way!

But did not we mention GET requests earlier? Now, we can turn them on, just open up routes.rb and add the following line:

get "/graphql", to: "graphql#execute"

We also need to slightly change the initialization of apollo-link-persisted-queries, which should send queries without query string as GET, and full queries as POST:

const link = createPersistedQueryLink({ useGETForHashedQueries: true }).concat(
  createHttpLink({ uri: "/graphql" })
);

Persisted queries can help you reduce the amount of traffic between your GraphQL server and its clients by storing query strings at the back-end. If you have the pro version of graphql-ruby–you're all set, otherwise, take a look at the open-source alternative graphql-ruby-persisted_queries we have just cooked up.

Read more dev articles on https://evilmartians.com/chronicles!

GitHub Actions: First impressions

Vladimir Dementyev — Fri, 06 Sep 2019 10:16:33 +0000

GitHub Actions are coming on strong—in my team, almost everyone who has applied for a beta program, me included, had recently got access to GitHub's latest "killer feature" that threatens to make life harder for Travis CI and CircleCI: Continuous Integration with GitHub Actions. Here are my first impressions.

// Detect dark theme var iframe = document.getElementById('tweet-1164301670429409281-5'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1164301670429409281&theme=dark" }

For the ultimate test, I decided to reuse the documentation generator setup from my previous article: "Keeping OSS documentation with Docsify, Lefthook, and friends"). To lint a documentation website for AnyCable, I used Lefthook locally and CircleCI for production. For my next documentation project, the one for TestProf, I decided to move all the CI functionality to GitHub Actions (sorry, Travis and CircleCI, of course I still love you).

Check the PR with the final implementation: palkan/test-prof#156

Warming up: dealing with the stale issues

The first thing I found impressive about GitHub Actions is that they could be used not only to deal with code pushes and pull requests but also react on other GitHub events or run on schedule!

One of the actions that GitHub offers you when you first open the "Actions" tab of your project is Stale: it allows you to mark issues and pull requests with a "stale" label and close them. That's what I usually did by hand (and was hoping to automate with the help of the Stale GitHub App).

It took me a few minutes to add this action:

# .github/workflows/stale.yml
name: Mark stale issues

on:
  schedule:
  - cron: "0 * * * *"

jobs:
  stale:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/stale@v1
      with:
        repo-token: ${{ secrets.GITHUB_TOKEN }}
        stale-issue-message: >
          ⚠ Marking this issue as stale since there has been no activity in the last 30 days.
          Remove stale label or comment or this issue will be closed in 15 days ⌛️
        stale-issue-label: stale
        days-before-stale: 30
        days-before-close: 15

After some time, I received a lot of notifications—the action took action:

I quickly realized that it wasn't a good idea: GitHub marked all the issues intentionally kept open (for discussion) as stale and spammed all the participants with the comment notification. I'm sorry, guys 😿.

It turned out that there is no ignore mechanism (e.g., by labels). There is one now, but still be careful: it is easy to get burned.

Migrating Markdown linters

Reimplementing the Markdown linting configuration I've already had for CircleCI was a pretty straightforward task:

name: Lint Docs

on:
  push:
    branches:
    - master
    paths:
    - "**/*.md"
  pull_request:
    paths:
    - "**/*.md"

jobs:
  markdownlint:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Ruby 2.6
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.6.x
    - name: Run Markdown linter
      run: |
        gem install mdl
        mdl docs
  rubocop:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Ruby 2.6
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.6.x
    - name: Lint Markdown files with RuboCop
      run: |
        gem install bundler
        bundle install --gemfile gemfiles/rubocop.gemfile --jobs 4 --retry 3
        bundle exec --gemfile gemfiles/rubocop.gemfile rubocop -c .rubocop-md.yml
  forspell:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Install Hunspell
      run: |
        sudo apt-get install hunspell
    - name: Set up Ruby 2.6
      uses: actions/setup-ruby@v1
      with:
        ruby-version: 2.6.x
    - name: Run Forspell
      run: |
        gem install forspell
        forspell docs/
  liche:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v1
    - name: Set up Go
      uses: actions/setup-go@v1
      with:
        go-version: 1.12.x
    - name: Run liche
      # see https://github.com/actions/setup-go/issues/14
      run: |
        export PATH=$PATH:$(go env GOPATH)/bin
        go get -u github.com/raviqqe/liche
        liche -r docs -d docs

A few noticeable differences compared to a CircleCI config:

Ability to trigger actions only when matching files have changed (See the paths: ["**/*.md"] declaration).
Inability to share the setup (checkout, dependency installation) between the jobs from the same action; there is no cache, or anything akin to CircleCI "workspaces".

Adding Neo and Trinity for RSpec

Adding the "Lint Docs" action went smoothly, so I decided to continue with migrating RSpec tests.

I'm running gems tests against multiple Ruby and frameworks versions to make sure most of the users are covered. I've been doing this successfully for years with Travis' Build Matrix feature:

matrix:
  fast_finish: true
  include:
    - rvm: 2.6.3
      gemfile: gemfiles/railsmaster.gemfile
    - rvm: jruby-9.2.7.0
      gemfile: gemfiles/jruby.gemfile 
    - rvm: 2.6.3
      gemfile: gemfiles/activerecord6.gemfile
    - rvm: 2.6.3
      gemfile: Gemfile
    - rvm: 2.5.3
      gemfile: Gemfile
    - rvm: 2.4.2
      gemfile: gemfiles/default_factory_girl.gemfile
    - rvm: 2.4.2
      gemfile: gemfiles/rspec35.gemfile
    - rvm: 2.4.2
      gemfile: gemfiles/activerecord42.gemfile
   allow_failures:
    - rvm: 2.6.2
      gemfile: gemfiles/railsmaster.gemfile
    - rvm: jruby-9.2.7.0
      gemfile: gemfiles/jruby.gemfile

GitHub Actions have a similar feature—strategy.matrix. And it's even stated in the documentation that the include-only variant works the same way as in Travis. However, that's not exactly true yet.

So, I had to take a detour and use the exclude option:

strategy:
  matrix:
    ruby: ["2.5.x", "2.6.x", "2.4.x"]
    gemfile: [
       "gemfiles/railsmaster.gemfile",
       "gemfiles/activerecord6.gemfile",
       "gemfiles/activerecord42.gemfile",
       "gemfiles/default_factory_girl.gemfile",
       "gemfiles/rspec35.gemfile"
    ]
    exclude:
    - ruby: "2.6.x"
      gemfile: "gemfiles/activerecord42.gemfile"
    - ruby: "2.6.x"
      gemfile: "gemfiles/rspec35.gemfile"
    - ruby: "2.6.x"
      gemfile: "gemfiles/default_factory_girl.gemfile"
    - ruby: "2.5.x"
      gemfile: "gemfiles/railsmaster.gemfile"
    - ruby: "2.5.x"
      gemfile: "gemfiles/activerecord42.gemfile"
    - ruby: "2.5.x"
      gemfile: "gemfiles/rspec35.gemfile"
    - ruby: "2.5.x"
      gemfile: "gemfiles/default_factory_girl.gemfile"
    - ruby: "2.4.x"
      gemfile: "gemfiles/railsmaster.gemfile"
    - ruby: "2.4.x"
      gemfile: "gemfiles/activerecord6.gemfile"

This configuration works exactly the same as the one from Travis (except the missing JRuby, we'll discuss it later), but unfortunately is much less readable.

Another thing I'd like to point out is that with GitHub Actions (compared to Travis) you have to deal with setting up a correct Gemfile yourself:

- name: Configure Gemfile
  run: |
    bundle config --global gemfile ${{ matrix.gemfile }}

Also, there is no allow_failures option. There is a steps.continue-on-failure toggle which could be used to achieve something similar, but with the strategy matrix.

Dealing with JRuby

You cannot use JRuby with the actions/setup-ruby action (or you can, but I couldn't find how?). It requires special treatment.

Here I applied another GitHub Actions feature—ability to use Docker containers to perform actions. That makes it works very similar to CircleCI.

I took the official JRuby image and added a separate job:

rspec-jruby:
  runs-on: ubuntu-latest
  container:
    image: jruby:9.2.8
    env:
      BUNDLE_GEMFILE: gemfiles/jruby.gemfile
  steps:
  - uses: actions/checkout@v1
  # I need git 'cause I use `git ls-files` in my .gemspec
  # to generate the list of the gem's files
  - name: Install git
    run: |
      apt-get update
      apt-get install -y --no-install-recommends git
  - name: Install deps and run RSpec
    run: |
      gem install bundler
      bundle install --jobs 4 --retry 3
      bundle exec rspec

And it just works!

Bonus: multiple badges

After merging the PR I've started looking for a way to add a build status badge to Readme (since I remove the old one from Travis).

The answer was found on Reddit pretty soon: https://github.com/{github_id}/{repository}/workflows/{workflow_name}/badge.svg.

What's interesting here is that you have a separate badge for each workflow. That means that, for example, you shouldn't be afraid of a "red" status due to the yet-another minor RuboCop release.

Another useful application of this "feature" is an ability to show that your library supports specific runtimes. For example, I split rspec workflow into two parts: one for different MRI version and another for JRuby. Now it's clear from the README that TestProf has been tested on JRuby and it's (hopefully) green!

GitHub Actions look very promising, especially for open source projects. The lack of some features (e.g., caching) would stop from using it as the primary CI/CD tool for commercial projects for now.

But that is just the beginning; we will see what's coming in the final release!

Read more dev articles on https://evilmartians.com/chronicles!

Keeping OSS documentation with Docsify, Lefthook, and friends

Vladimir Dementyev — Mon, 02 Sep 2019 18:28:51 +0000

"A program is only as good as its documentation."—Joe Armstrong, the author of the Erlang programming language

What makes a good open source project? If you are into Ruby, you can check out all the best practices in one place at Gem Check (created by yours truly). But even regardless the language or the stack—one thing is vital for all OSS projects: documentation.

Kelsey Hightower

@kelseyhightower

So you built it. Nobody's using it. Did you forget the docs? Aha!

01:08 AM - 22 Aug 2017

130 440

All open source projects, independently of the size, must be documented (even the now-deprecated, infamous left-pad is documented). In most cases, a well-crafted README is more than enough, but you can go further and create an "awesome readme".

However, as the project grows, README-backed documentation stops working. We might say "~~Rails~~ README doesn't scale!". And we will be right.

This brings us to a question: "What scales better than a single markdown page?"

In this text, I am sharing my answer to that.

In the Beginning, There Was Chaos

Before we start talking about docsify, let me show you some other tools I used, before settling on the one and only.

GitHub Wiki

My first project to outgrow its README was AnyCable. Without giving it much thought, I started moving bits of documentation into the GitHub's built-in wiki (it's still there, for older versions of the gem). If we already have the tool—that's the way to go, right? Well, not necessarily.

It turned out that the fact that GitHub wiki is built-in is the only advantage, while the list of cons drags on:

Updating code and docs independently is likely to lead to inconsistency.
Web editor has much to be desired; you cannot easily upload images, for example (technically it is possible, but it requires cloning the wiki repo).
Cross-references are easy to break just by changing a title; there is no way to have a permanent URL (or I was missing something).

`docs` folder

One way to resolve all the Wiki weak points (see the pun?) at once is to create a docs folder in the repo and populate it with markdown files. GitHub displays .md files with formatting when you open them in your browser. And you can also edit the contents right from the web UI! If that's the case, why use the wiki feature at all?

So, we found a good way to store the documentation contents. But what about the UI/UX? Don't we want to make our documentation more user-friendly (for example, add searching functionality)? Yes, we do.

Let's convert our GitHub-driven docs to web format.

Jekyll & GitHub Pages

GitHub helps with setting up a documentation website backed by Jekyll in a few clicks: go to "Settings" -> "GitHub Pages," choose the source for the website (we use the docs folder) and pick a Jekyll theme to your liking.

Now visit the URL from the Settings page and find your brand new documentation website there!

Unfortunately, the GitHub Pages Jekyll integration is limited, especially in terms of the available plugins. You cannot go far with it. And, in my opinion, Jekyll is a bit too complicated if you want to customize the looks of your page or add some interactivity.

Let's check out modern tools.

Docusaurus

The first modern documentation generator I have tried was Docusaurus. We built the Clowne's gem documentation with it.

But I have to admit that the experience was not so pleasant:

Given the fact that the library is built with React, I expected it to have more straightforward customization options. However, you don't really get access to internals and library authors don't want you to do the serious tweaking.
At the time of my first usage, it did not have the support for live reload, which is crucial for local development (now it seems to be fixed).
It requires a separate building step (yarn run build).

So I started looking for other options and found docsify.

Docsifying documentation

Docsify uses a different approach, compared to Jekyll or Docusaurus, to "generating" a website: it renders markdown files on the fly, and does not require a build phase.

There is also support for Server-side rendering and even for offline mode.

To "docsify" your docs, you need to do the following:

Add docs/.nojekyll file to disable Jekyll.
Add index.html that loads and configure docsify:

<!DOCTYPE html>
<html>
<head>
  <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <meta charset="UTF-8">
  <link rel="stylesheet" href="//unpkg.com/docsify/themes/vue.css">
</head>
<body>
  <div id="app"></div>
  <script>
    window.$docsify = {
      loadSidebar: true,
      subMaxLevel: 2,
      repo: 'palkan/docs-example',
      basePath: '/docs-example/',
      auto2top: true,
      homepage: 'https://raw.githubusercontent.com/palkan/docs-example/master/README.md'
    }
  </script>
  <script src="//unpkg.com/docsify/lib/docsify.min.js"></script>
  <script src="//unpkg.com/prismjs/components/prism-bash.min.js"></script>
  <script src="//unpkg.com/prismjs/components/prism-ruby.min.js"></script>
</body>
</html>

And that is it! Now you have something like this:

Note that I have added some specific configuration options:

basePath: '/docs-example/ defines the root path of your website (which is the repo name for personal projects on GitHub Pages);
homepage: '...' is set to the repo's README (by default docsify uses the docs/README.md file); that allows us to keep both home pages (GitHub and web) in sync.

And that's just the beginning! One of the main advantages of docsify is the simplicity of adding useful features via plugins.

Let's add a searching functionality.

All we need is to add these two lines of code:

 window.$docsify = {
   loadSidebar: true,
   subMaxLevel: 2,
+  search: 'auto',
   repo: 'palkan/docs-example',
   basePath: '/docs-example/',
   auto2top: true,
   homepage: 'https://raw.githubusercontent.com/palkan/docs-example/master/README.md'
   }
 </script>
 <script src="//unpkg.com/docsify/lib/docsify.min.js"></script>
 <script src="//unpkg.com/docsify/lib/plugins/search.min.js"></script>
 <script src="//unpkg.com/prismjs/components/prism-bash.min.js"></script>
 <script src="//unpkg.com/prismjs/components/prism-ruby.min.js"></script>

And voilà! We can search through our documentation! The searching is implemented on the client side and is backed by indexes saved to localStorage.

Another thing I like about docsify is the ease of customizing styles: the library uses CSS properties, thus making it possible to change colors and layouts without building your own CSS!

You can change the color and font sizes right in your index.html:

<style>
  :root {
    --theme-color: #ff5e5e;
    --theme-color-light: #fd7373;
    --theme-color-dark: #f64242;
    --theme-color-secondary: #ff5e5e;
    --theme-color-secondary-dark: #f64242;
    --theme-color-secondary-light: #fd7373;
    --text-color-base: #363636;
    --text-color-secondary: #646473;
  }
</style>
<body>
  ...
</body>

Isn't it awesome?

The code for the example above could be found on GitHub: palkan/docs-example.

Check out AnyCable documentation website and the corresponding repo for a more advanced example!

Bonus*: you can find the implementation of the floating action button for "Edit on GitHub" functionality in this gist.

Keeping docs healthy with linters

In the second part of this tutorial, I would like to share my approach to keeping documentation in a healthy state. And by the "healthy state," I mean:

Style consistency for source files (Markdown) and code examples (Ruby).
Correct spelling.
Valid code examples (from the syntax point of view).
Valid links (no link should lead to 4xx/5xx).

It's not surprising that for all of the above there is an open source tool (and sometimes many).

Markdownlint helps me to enforce Markdown files style (there is also a NodeJS version and a VS Code plugin).

I also usually disable a couple of rules:

Line length (MD013)—modern editors (such as VS Code) could handle this by wrapping long lines.
HTML fragments—sometimes Markdown is not enough.

To do that I put a .mdlrc file in the project's root with the following contents:

rules "~MD013", "~MD033"

To deal with Ruby syntax I use RuboCop along with the rubocop-md plugin that I wrote specifically for this task. As a default style configuration, I've recently started using standard.

To make this setup work you need to:

Install standard and rubocop-md gems (gem install standard and gem install rubocop-md)
Add a .rubocop.yml with the following contents:

require:
  - standard/cop/semantic_blocks
  - rubocop-md

inherit_gem:
  standard: config/base.yml

Standard/SemanticBlocks:
  Enabled: false

Run RuboCop.

For spellchecking, there is yet another Ruby tool—Forspell. It is a wrapper over a well-known Hunspell package.

Due to the number of technical terms, you may see a lot of warnings from Forspell during the first run:

$ forspell docs/

docs/development/lefthook.md:5: lefthook (suggestions: left hook, left-hook, leftmost)
docs/development/lefthook.md:9: lefthook (suggestions: left hook, left-hook, leftmost)
docs/development/lefthook.md:11: Hombrew (suggestions: Hombre, Hombres, Hombre w)
docs/development/lefthook.md:17: Golang (suggestions: Golan, Golan g, Angolan)

That could be easily fixed by running Forspell with the --gen-dictionary flag: it generates a forspell.dict file with all the unknown words. Don't forget to scan this file with your eyes and remove the actual typos.

Finally, to make sure that our documentation does not have any broken links, I use liche—a link checker for Markdown and HTML written in Go:

$ liche -r docs/

 ERROR    https://githb.com/palkan/anyway_config
                Dialing to the given TCP address timed out

Liche lacks some features I wish it had: for example, it does not warn about URLs responding with 404. Nevertheless, I found it a bit better than other existing tools.

In order to manage all these linters, I use Lefthook for local development and CircleCI for pull requests.

Here is the contents of my lefthook.yml, a file that stores Lefthook's configuration:

pre-commit:
  commands:
    mdl:
      glob: "**/*.md"
      run: mdl {staged_files}
    liche:
      glob: "**/*.md"
      run: liche -r docs
    forspell:
      glob: "**/*.md"
      run: forspell {staged_files}
    rubocop:
      glob: "**/*.md"
      run: rubocop {staged_files}

CirclCI configuration is a little bit more verbose, but does pretty much the same:

version: 2.1

workflows:
  version: 2
  build_and_test:
    jobs:
      - checkout
      - md_lint:
          requires:
            - checkout
      - links_lint:
          requires:
            - checkout
      - spelling:
          requires:
            - checkout
      - rubocop:
          requires:
            - checkout

executors:
  golang:
    docker:
      - image: circleci/golang:1.12.4-stretch
  ruby:
    docker:
      - image: circleci/ruby:2.5-stretch

jobs:
  checkout:
    executor: ruby
    steps:
      - restore_cache:
          keys:
            - project-source-v1-{{ .Branch }}-{{ .Revision }}
            - project-source-v1-{{ .Branch }}
            - project-source-v1
      - checkout
      - save_cache:
          key: project-source-v1-{{ .Branch }}-{{ .Revision }}
          paths:
            - .git
      - persist_to_workspace:
          root: .
          paths: .
  md_lint:
    executor: ruby
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Install mdl
          command: gem install mdl
      - run:
          name: Markdown lint
          command: mdl docs
  links_lint:
    executor: golang
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Install liche
          command: go get -u github.com/raviqqe/liche
      - run:
          name: Check links
          command: liche -r docs
  spelling:
    executor: ruby
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Install hunspell
          command: sudo apt-get install hunspell
      - run:
          name: Install forspell
          command: gem install forspell
      - run:
          name: Check spelling
          command: forspell docs/
  rubocop:
    executor: ruby
    steps:
      - attach_workspace:
          at: .
      - run:
          name: Install standard
          command: gem install standard
      - run:
          name: Install rubocop-md
          command: gem install rubocop-md
      - run:
          name: Check Ruby style
          command: rubocop

The complete example can be found in the docs.anycable.io repo (see PRs #14 and #15).

It took me a while to come with this setup (literally, years). Now I can spin up a new documentation website in minutes. Hope you found this article useful and will consider using a similar approach next time you need web-based docs for your projects.

Documentation for the win!

Read more dev articles on https://evilmartians.com/chronicles!

Better web video with AV1 codec

Andrey Sitnik — Tue, 20 Aug 2019 16:46:59 +0000

Learn how to instantly improve online viewing experience for your users by embracing the new AV1 video format that is already supported by Chrome and Firefox. This short guide will also show how to replace your GIF's with videos, using AV1 or H.264, to make your files twenty to forty times smaller.

The bets are placed. Both YouTube and Netflix have named AV1 a video codec for the future: Google's video service is already using it on TestTube (new, experimental features for YouTube). Netflix has been calling AV1 "our primary next-gen codec" for a while now. At Evil Martians, we have already tried AV1 at our landing page and at the landing page of Amplifr. In this article, we will share our experience with a new video format and give step-by-step instructions for optimal encoding strategies.

Codecs and Containers

With static images, you don't have to think twice: opt for JPEG or PNG supported by all browsers, or experiment with a more compact Google-developed WebP for newer browsers. You can almost always (barring some dirty hacking tricks that tools like imgproxy can protect you against) be sure that an image file with .png extension is, indeed, a PNG.

With video files, it is a bit more tricky than that. File extensions (.mp4, .wmv, .webm or .mov) barely represent containers, up to three different formats are used to make a video file happen:

Video codec: determines the compression strategy for your video, this is where the trade-offs are made between quality and quantity. On the web, some popular video codecs are H.264, HEVC, VP9, and now AV1.
Audio codec: does the same for audio. If your video does not have sound, you can do without it. Otherwise, the popular choices are MP3, Opus, and AAC.
Containers store both video (compressed by some video codec) and audio streams (compressed by some audio codec), and can also add extra details like subtitles and meta information. Popular containers are MP4, MOV, WebM.

So, when you see .mp4 extension, the only thing you can be sure about is that the MP4 container had been used to package a file. The choice of codecs depends entirely on a creator: it can be H.264/AAC, or AV1/Opus, or something else.

Meet AV1

AV1 is a video codec that was first released almost a year ago: in March 2018. It is designed to compete with previous codec generations such as HEVC/VP9 and H.264/VP8.

Video codecs generations diagram by Tsahi Levent-Levi (source)

To get familiar with technologies used in a new generation video code, feel free to read "Introducing AV1" and "AV1: next generation video".

With all the low-level trickery involved, AV1 is capable of generating files that are up to 30-50% smaller than H.264/VP8 and up to 30% smaller than HEVC, even though, due to being still mostly experimental, it has some problems (at the time of this writing):

The encoder is not optimized yet. As a result, encoding is extremely slow (an upcoming encoder written in Rust attempts to solve this issue). The format is not yet ready for live streaming. However, it is perfectly suitable for web, as your average landing page will usually have a short embedded video that rarely changes.
While supported by Chrome and Firefox, AV1 lacks implementations for Safari and Edge (although Microsoft already has AV1 support in early beta). So you need to have at least two versions of all your videos: AV1 for Chrome/Firefox and H.264 for everything else. Ideally, you should have a third, HEVC version for your Safari users on desktop and mobile and we will show how to prepare all three of those files.

The central promise for of AV1 is maintaining high image quality even at low bitrates, thus allowing for smaller files without apparent compression artifacts.

A chart by Jan Ozer plots data rate against the VMAF quality metric. AV1 is a clear winner.

How to use AV1 right now

Now we are going to show a sequence of steps required to produce the quality video content for the web with AV1. First, you need to choose a container: in theory, it does not matter, but MP4 is recommended and seems to be the most popular at the moment. For the audio codec, we will use Opus with AV1 as an efficient and free alternative.

To ensure the best cross-browser compatibility, we will produce not one, but three files:

For desktop Chrome and Firefox (31% of users as of August 2019): MP4 container with AV1 video codec and Opus audio codec.
For Safari and Edge (16% of users): MP4 with HEVC and AAC.
For other browsers: a larger file in MP4 container with H.264 for video and AAC for audio.

You can also go with just options 1 and 3, you will still ensure that all your users can see the video.

For conversion, I recommend using FFmpeg in a terminal. There are plenty of GUI tools for video compression, but CLI allows for steps that are easily reproducible and can be automated with a script. Make sure that you are using the most recent version of FFmpeg, as versions below 4.1 do not support AV1 in an MP4 container. Here are the steps to install it.

For Mac:

Make sure you have Homebrew.
brew install ffmpeg

For Linux, we recommend using the latest build of FFmpeg from the official website, as at the time of this writing not all package managers contain most recent, AV1-enabled versions:

wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
tar -xf ffmpeg-release-amd64-static.tar.xz
sudo cp ffmpeg-4.1-64bit-static/ff* /usr/local/bin/

For Windows, you can use this guide.

Once ffmpeg executable is available in your command line, let's generate the H.264 file (to ensure compatibility with older browsers). Since all our files will use MP4 as a container, I will use .av1.mp4, .hevc.mp4, and .h264.mp4 file extensions. Here is the command you will need to use (don't worry, we will walk you through all the options in a moment).

# Replace SOURCE.mov with a path to your source video file

ffmpeg -i SOURCE.mov -map_metadata -1 -c:a libfdk_aac -c:v libx264 -crf 24 -preset veryslow -profile:v main -pix_fmt yuv420p -movflags +faststart -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" video.h264.mp4

Now open the resulting video.h264.mp4 file and check your quality. If you are satisfied with the result, but the size still seems too big, try adjusting the -crf option (try -crf 26 or -crf 28): it will reduce the file size, but also the quality, so try to find an acceptable trade-off. That process, frankly, is more art than science.

You can convert H.264 to AV1 if you don't have a single uncompressed source.

Now it is time to generate the AV1 file. The command below will take longer than the one for H.264, but that is to be expected: at the moment, AV1 codec does not use the full power of CPU. That is a curse, but also a blessing: if you are about to encode several files at the same time, it is safe to do so, as multi-threading is currently not supported by default.

ffmpeg -i SOURCE.mov -map_metadata -1 -c:a libopus -c:v libaom-av1 -crf 34 -b:v 0 -pix_fmt yuv420p -movflags +faststart -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -strict experimental video.av1.mp4

Play around with the -crf setting for an optimal size/quality balance.

Now the same for HEVC:

ffmpeg -i SOURCE.mov -map_metadata -1 -c:a libfdk_aac -c:v libx265 -crf 24 -preset veryslow -pix_fmt yuv420p -movflags +faststart -tag:v hvc1 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" video.hevc.mp4

Then copy all three resulting files (video.h264.mp4, video.hevc.mp4, and video.av1.mp4) to the root of your web project.

Understand compression options

For now, the commands above look like black magic spells, but all those keys are used for a purpose. Here is what they do:

-i SOURCE.mov sets the source video file for input. FFmpeg will take video and audio streams from this file, convert them, and pack into a new container.
-map_metadata -1 will remove video metadata (like the name of a tool that was used initially to create a video). Sometimes metadata is useful, but for web development it is rarely the case.
-c:a libopus or -c:a libfdk_aac selects an audio codec.
-c:v libaom-av1 selects a video codec, a library to compress images into a video stream.
-crf 34 stands for Constant Rate Factor and sets your size/quality balance. Think of it as the quality slider for JPEG, but it goes in the opposite direction (0 stands for best quality and bigger size). CRF scale is different for H.264 and AV1: H.264 goes from 0 to 51, AV1 from 0 to 61. As a result, you will have different CRF ratios for AV1 and H.264. According to this guide by Facebook, here are the optimal mappings between H.264 and AV1 CRF values: 19 → 27, 23 → 33, 27 → 39, 31 → 45, 35 → 51, 39 → 57.
-preset veryslow forces H.264 and HEVC codecs to generate smaller video file even if it will be much longer.
-profile:v main that we use in our H.264 command selects the video codec profile. We can only use "Main", as our video will not be played in Safari otherwise.
-b:v 0 a sets minimum bitrate to force the constant quality mode in AV1.
-pix_fmt yuv420p (pixel format) is a trick to reduce the size of a video. Basically, it uses full resolution for brightness and a smaller resolution for color. It is a way to fool a human eye, and you can safely remove this argument if it does not work in your case.
-movflags +faststart moves the important information to the beginning of the file. It allows browser to start playing video during downloading.
-tag:v hvc1 enables native HEVC video support on Apple operating systems.
-vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" is a way to ensure the produced video will always have an even size (some codecs will only work with sizes like 300x200 and 302x200, but not with 301x200). This option tells FFmpeg to scale the source for the closes even resolution. If your video dimensions were even in the first place, it would not do anything.
-strict experimental option needs to be used for AV1 as AV1 encoder is still experimental.
video.av1.mp4 sets the name for the output file.

Play nice with browsers

Now you need to make sure that browsers will display the right file depending on whether it is supported or not. Luckily, we can set a type attribute on a source element, and only the supported file will be played. For more <video> tag options, look here.

<video controls width="600" height="400">
  <source
    src="video.hevc.mp4"
    type="video/mp4; codecs=hevc,mp4a.40.2"
  />
  <source
    src="video.av1.mp4"
    type="video/mp4; codecs=av01.0.05M.08,opus"
  />
  <source
    src="video.h264.mp4"
    type="video/mp4; codecs=avc1.4D401E,mp4a.40.2"
  />
</video>

Source tags work similarly to if...else statements: the browser will read the list of <source> tags top to bottom and play the first one with supported video type.

Type attribute describes a file format: which container (video/mp4 for MP4), video codec (av01.0.05M.08 for AV1, hevc for HEVC and avc1.4D401E for H.264), and audio codec (opus for Opus and mp4a.40.2 for AAC) should be used.

Bonus: how to convert GIFs to AV1 and H.264

In modern times, using GIF for video fragments is a poor practice. GIFs take 20 to 40 times more space than H.264 or AV1. They also require more CPU and power and drain more battery than modern video formats. If you need short animation sequences on your website in 2019, always opt for video codecs. Luckily, FFmpeg supports GIF files as an input source.

Here's how to convert your GIF to H.264:

ffmpeg -i IMAGE.gif -map_metadata -1 -an -c:v libx264 -crf 24 -preset veryslow -profile:v main -pix_fmt yuv420p -movflags +faststart -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" animation.h264.mp4

And here's how to go even further and convert it to AV1:

ffmpeg -i IMAGE.gif -map_metadata -1 -an opus -c:v libaom-av1 -crf 50 -b:v 0 -pix_fmt yuv420p -movflags +faststart -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -strict experimental animation.av1.mp4

Now we can use animation.h264.mp4 and animation.av1.mp4 in our HTML. Just replace VIDEO_WIDTH, VIDEO_HEIGHT, and PATH_TO_VIDEO:

<video
  autoplay
  loop
  muted
  playsinline
  width="VIDEO_WIDTH"
  height="VIDEO_HEIGHT"
>
  <source
    src="PATH_TO_VIDEO/animation.av1.mp4"
    type="video/mp4; codecs=av01.0.05M.08"
  />
  <source
    src="PATH_TO_VIDEO/animation.h264.mp4"
    type="video/mp4"
  />
</video>

autoplay and loop attributes will emulate the expected behavior of a GIF: looping an animation after the website was loaded. playsinline will forbid Safari from opening the video in full-screen mode.

And that concludes our practical guide!

Even though AV1 codec is still considered experimental, you can already leverage its high-quality, low-bitrate features for a sizable chunk for your web audience (users with current versions of Chrome and Firefox). Of course, you would not want to leave users for other browsers hanging, but the attributes for <video> and <source> tags make implementing this logic easy, and in pure HTML, you don't need to go at length to detect user agents with JavaScript. Mastering a few FFmpeg commands also seems like an easy way to improve the video viewing experience for your visitors. We already use AV1 in production on a couple of our projects and have not encountered any significant problems (except for video compression times, but, again, we are dealing mostly with short static sequences).

Fullstaq Ruby: First impressions, and how to migrate your Docker/Kubernetes Ruby apps today

Andrey Novikov — Wed, 07 Aug 2019 14:15:13 +0000

What is Fullstaq Ruby?

Fullstaq Ruby is a custom build of standard MRI Ruby interpreter with memory allocator replaced, security patches applied, and more goodies on the way.

If some old timers are here, they can remember REE–Ruby Enterprise Edition–from ancient times of Ruby 1.8.7 and Ruby on Rails 2.2 (almost ten years ago!). Ah, good ol’ times! You could install it via RVM or Rbenv, and some legacy applications are still running on it or have been just recently migrated. REE has a dozen of different patches on top of Ruby 1.8.7 to improve performance, reduce memory consumption, adjust obsolete security settings, and so on.

In MRI 1.9.x most of these problems were solved, and, as it gained adoption, REE became obsolete. But even modern “vanilla” MRI still has quirks that can be fixed relatively easy. The most annoying of them is memory bloat due to memory fragmentation.

So it is not at all surprising that Hongli Lai, the creator of REE, have released Fullstaq Ruby.

REE is dead, long live Fullstaq Ruby!

Why do we need it?

At one of our projects at Evil Martians we were experiencing severe memory bloat. Our application does a lot of IO, and we have a lot of Sidekiq processes with high concurrency setting (20 threads per process). This setting is optimal from the performance point of view because workers are mostly making requests to different remote APIs, our own database, and caches. But such a high level of concurrency also leads to high memory fragmentation. Our Sidekiq processes eat several gigabytes of RAM each.

Read more about choosing Sidekiq concurrency setting in the Sidekiq in Practice part 1 by Nate Berkopec.

We have decided to replace our MRI 2.6.3 to Fullstaq Ruby 2.6.3 with jemalloc to see how it will perform.

Now that's the difference!

We tried Fullstaq Ruby on a commercial application that runs in production and serves requests from paying clients around the clock.

First of all: nothing broke. Zero downtime!

Now, take a look at these monitoring graphs. Memory bloat of long-running processes has practically gone!

Web application processes have become very stable in memory consumption (4 times less memory!). Bloat still occurs sporadically, but still the readings show that about 50% less memory is consumed during spikes.
Background job workers (we are using Sidekiq) also lost two thirds of their weight. From 1.5-2 GB before to 500-700 MB after the migration to Fullstaq Ruby.
There is no noticeable difference in memory consumption for short processes (e.g., cron jobs)
We didn't notice any changes in response times or CPU utilization.

The graphs above prove that memory fragmentation was the reason for high memory consumption.

And that’s it—quite an improvement for swapping one ruby binary for another, isn’t it?

Alternatives?

If jemalloc isn't an option for you or you cannot afford to replace MRI with something else, you can try MALLOC_ARENA_MAX=2 spell to adjust MRI's standard glibc malloc behavior. Results will be close enough to Fullstaq Ruby to treat them almost as equal.

In our case, Ruby with limited number of malloc arenas (on the right) consumed about 50-100 MB more memory than Ruby with jemalloc (on the left).

Read more and see benchmarks of MALLOC_ARENA_MAX=2 in this post:

Cables vs. malloc_trim, or yet another Ruby memory usage benchmark

Vladimir Dementyev ・ Mar 19 '19 ・ 6 min read

#ruby #benchmarks #rails

We decided to stick with Fullstaq Ruby.

How to install?

At the moment the only way to install Fullstaq Ruby is to use deb or rpm packages (either installing directly or via repositories). But we deploy our app to Kubernetes cluster, so we need a Docker image. As there is no "container edition" available on the official website yet, so let’s build our own image–actually, it is not that hard!

Let's use Debian 9, as this is the Linux distribution being used by official Ruby Docker image, and define the Ruby version:

FROM debian:stretch-slim

ARG RUBY_VERSION=2.6.3-jemalloc

And then install prerequisites, add Fullstaq Ruby APT repository, install Ruby itself and cleanup apt caches—all in a single command to reduce the Docker layer size:

RUN apt-get update -q \
    && apt-get dist-upgrade --assume-yes \
    && apt-get install --assume-yes -q --no-install-recommends curl gnupg apt-transport-https ca-certificates \
    && curl -SLf https://raw.githubusercontent.com/fullstaq-labs/fullstaq-ruby-server-edition/master/fullstaq-ruby.asc | apt-key add - \
    && echo "deb https://apt.fullstaqruby.org debian-9 main" > /etc/apt/sources.list.d/fullstaq-ruby.list \
    && apt-get update -q \
    && apt-get install --assume-yes -q --no-install-recommends fullstaq-ruby-${RUBY_VERSION} \
    && apt-get autoremove --assume-yes \
    && rm -fr /var/cache/apt

Fullstaq Ruby also installs Rbenv as dependency but we don't need it in Docker, so let's add ruby and gems binaries to system $PATH in the same way that official Docker image for Ruby does:

ENV GEM_HOME /usr/local/bundle
ENV BUNDLE_PATH="$GEM_HOME" \
    BUNDLE_SILENCE_ROOT_WARNING=1 \
    BUNDLE_APP_CONFIG="$GEM_HOME" \
    RUBY_VERSION=$RUBY_VERSION \
    LANG=C.UTF-8 LC_ALL=C.UTF-8

# path recommendation: https://github.com/bundler/bundler/pull/6469#issuecomment-383235438
ENV PATH $GEM_HOME/bin:$BUNDLE_PATH/gems/bin:/usr/lib/fullstaq-ruby/versions/${RUBY_VERSION}/bin:$PATH

CMD [ "irb" ]

And that's it!

We’ve already built and published this image. You can pull it from our repository at quay.io:

docker pull quay.io/evl.ms/fullstaq-ruby:2.6.3-jemalloc-stretch-slim

Dockerfiles are available on GitHub: https://github.com/evilmartians/fullstaq-ruby-docker

Now we can just replace base image in our application Dockerfile:

-ARG RUBY_VERSION=2.6.3
+ARG RUBY_VERSION=2.6.3-jemalloc

-FROM ruby:${RUBY_VERSION}-stretch-slim
+FROM quay.io/evl.ms/fullstaq-ruby:${RUBY_VERSION}-stretch-slim

And deploy it to staging and then to production.

Feel free to do the same!

Recap

Migration is smooth. Just reinstall Ruby and gems, and everything should just work.
Application servers and background jobs worker processes should reduce memory consumption drastically.
There is no noticeable difference in memory consumption for short processes (like cron jobs or scripts).
Performance may slightly improve, but it depends on your workload profile.

Speeding up Go Modules for Docker and CI

Sergey Ponomarev — Mon, 05 Aug 2019 15:19:53 +0000

Finally, the Golang world has a built-in, conventional dependency manager in the ecosystem: Go Modules. What began in Go 1.11 as an opt-in feature has become widely adopted by the community, and we are so close to Go 1.13 when Go Modules will be enabled by default. The delightful dilemma of choosing the “best” tool can be finally resolved.

I can’t help but mention two features which are very close to my heart:

– No more $GOPATH imprisonment! In my years of experience, I’d gotten used to storing everything I work on in ~/Projects/ and its subfolders somewhere in the home directory, no matter the programming language. So, being forced to keep my Golang stuff in another specific place and respect SCM url in the path was a real pain, and made routine cd operations feel like such a chore. No longer an issue!
– No more vendoring! Dependency updates don’t produce enormous PR diffs to read, and repositories are lighter. I can just remove the vendor folder from my source code and forget about it.

The migration to Go Modules is pretty simple and won’t take more than a couple of minutes, especially if you use any of the supported package managers to migrate from.

$ go mod init
$ rm vendor/*
$ go test ./...
$ git add .
$ git commit

That’s pretty much it!

With Go Modules your dependencies are not a part of your source code anymore. The toolchain downloads them on its own, keeps modules up-to-date, and caches them locally in $GOPATH/pkg/mod for future use. That sounds perfect for when all of your processes occur in a stateful environment like a laptop, but what about stateless builds in your CI pipeline or Docker? Every now and again Go will download every item in your dependencies and waste your priceless time. Let’s fix that with some caching!

Caching on CI

It’s such a common situation to cache dependencies between builds on CI that some of the services provide a simplified, ecosystem-specific syntax to make it easier. Alas, I haven’t found specific Go Modules caching on popular CIs yet, so let’s do it manually.

If you use TravisCI, it’s very straightforward. Just add those lines to your .travis.yml config and you’re all set:

cache:
  directories:
    - $GOPATH/pkg/mod

Setting up dependency caching on my favorite CircleCI is a little more verbose. Wrap go mod download or your build step in the code below. Golang will take care of the missing dependencies and CircleCI will cache them between builds relying on the content of the go.sum file.

      - restore_cache:
          keys:
            - go-modules-v1-{{ checksum "go.sum" }}
            - go-modules-v1
      # get dependencies here with `go mod download` or implicitly 
      # with `go build` or `go test`
      - save_cache:
          key: go-modules-v1-{{ checksum "go.sum" }}
          paths:
            - "/go/pkg/mod"

Here are the results of boosting of my little project on CircleCI:

Before:

`go test ./...` => 00:20s

After cache warm-up:

Restoring Cache => 00:03s
`go test ./...` => 00:06s
Saving Cache    => 00:00s

Not bad at all: 2x faster CI build, and for free.

Caching in Docker

There are two completely different use cases for how we use Docker: for the development process to isolate an application and its environment, and for packing production builds.

In Development

If you follow Test Driven Development (TDD) caching, Go Modules can significantly increase your development productivity. You definitely know how crucial it is to have as fast a test suite as possible.

Using Docker Compose, cache your modules in a separate volume, and see the performance boost. I saved 20 seconds. Not that bad for a small change!

Here is a minimal docker-compose.yml, simplified for the sake of brevity, with highlighted volumes changes:

version: '3'
services:
  app:
    image: application:0.0.1-development
    build:
      context: .
      dockerfile: docker/development/Dockerfile
    volumes:
      - .:/app
      - go-modules:/go/pkg/mod # Put modules cache into a separate volume
  runner:
    <<: *app
  test:
    <<: *app
    command: go test ./...
volumes:
  go-modules: # Define the volume

In production

For production builds, we can take advantage of the layer caching power. Dependencies change less often than the code itself—let’s make it a separate step in your Dockerfile before the build phase.

# `FROM` and other prerequisites here skipped for the sake of brevity

# Copy `go.mod` for definitions and `go.sum` to invalidate the next layer
# in case of a change in the dependencies
COPY go.mod go.sum ./
# Download dependencies
RUN go mod download

# `RUN go build ...` and further steps

In a nutshell

Introducing Go Modules was an exciting moment and a significant relief to the Golang community. It brought us a lot of excellent features we had been waiting a long time for. Don’t hesitate to try Modules if you haven’t yet. It’s pretty easy to migrate to it, but don’t forget to change your CI or Docker settings to avoid downloading overhead and keep your builds blazing fast.

Ruby on Whales: Dockerizing Ruby and Rails development

Vladimir Dementyev — Wed, 24 Jul 2019 19:27:13 +0000

Originally posted in Martian Chronicles.

This post is a b-side of my recent RailsConf talk "Terraforming legacy Rails applications" (video, slides).

In this post, I am not going to convince you to switch to Docker for application development (though you can check the RailsConf video for some arguments). My goal is to share the configuration I currently use for Rails projects, and which was born in ~~production~~ development at Evil Martians. Feel free to use it!

I've started using Docker in my development environment about three years ago (instead of Vagrant which was too heavy for my 4GB RAM laptop). It wasn't all roses since the start, of course—I spent two years trying to find a configuration that is good enough, suitable not only for myself but also for my team.

Let me present this config here and explain (almost) every line of it, because we've all had enough of cryptic tutorials that just assume you know stuff.

The source code could be found in the evilmartians/terraforming-rails repository on GitHub.

We use the following stack in this example:

Ruby 2.6.3
PostgreSQL 11
NodeJS 11 & Yarn (for Webpacker-backed assets compilation)

`Dockerfile`

Dockerfile defines the environment for our Ruby application: this is where we run servers, console (rails c), tests, Rake tasks, interact with our code in any way as developers:

ARG RUBY_VERSION
# See explanation below
FROM ruby:$RUBY_VERSION

ARG PG_MAJOR
ARG NODE_MAJOR
ARG BUNDLER_VERSION
ARG YARN_VERSION

# Add PostgreSQL to sources list
RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
  && echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list

# Add NodeJS to sources list
RUN curl -sL https://deb.nodesource.com/setup_$NODE_MAJOR.x | bash -

# Add Yarn to the sources list
RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo 'deb http://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list

# Install dependencies
# We use an external Aptfile for that, stay tuned
COPY .dockerdev/Aptfile /tmp/Aptfile
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    build-essential \
    postgresql-client-$PG_MAJOR \
    nodejs \
    yarn=$YARN_VERSION-1 \
    $(cat /tmp/Aptfile | xargs) && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    truncate -s 0 /var/log/*log

# Configure bundler and PATH
ENV LANG=C.UTF-8 \
  GEM_HOME=/bundle \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3
ENV BUNDLE_PATH $GEM_HOME
ENV BUNDLE_APP_CONFIG=$BUNDLE_PATH \
  BUNDLE_BIN=$BUNDLE_PATH/bin
ENV PATH /app/bin:$BUNDLE_BIN:$PATH

# Upgrade RubyGems and install required Bundler version
RUN gem update --system && \
    gem install bundler:$BUNDLER_VERSION

# Create a directory for the app code
RUN mkdir -p /app

WORKDIR /app

This configuration contains the essentials only and could be used as a starting point. Let me show what we are doing here.

The first two lines could look a bit strange:

ARG RUBY_VERSION
FROM ruby:$RUBY_VERSION

Why not just FROM ruby:2.6.3, or whatever Ruby stable version du jour it is? We want to make our environment configurable from the outside using Dockerfile as a sort of a template:

the exact versions of runtime dependencies are specified in the docker-compose.yml (see below);
the list of apt-installable dependencies is stored in a separate file (also see below).

The following three lines define arguments for PostgreSQL, NodeJS, Yarn, and Bundler versions:

ARG PG_MAJOR
ARG NODE_MAJOR
ARG BUNDLER_VERSION
ARG YARN_VERSION

Since we do not expect anyone to use this Dockerfile without Docker Compose, we do not provide default values.

Installing PostgreSQL, NodeJS, Yarn via apt requires adding their deb packages repos to the sources list.

For PostgreSQL (based in the official documentation):

RUN curl -sSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
  && echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' $PG_MAJOR > /etc/apt/sources.list.d/pgdg.list

For NodeJS (from NodeSource repo):

RUN curl -sL https://deb.nodesource.com/setup_$NODE_MAJOR.x | bash -

For Yarn (from the official website):

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo 'deb http://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list

Now it's time to install the dependencies, i.e. run apt-get install:

COPY .dockerdev/Aptfile /tmp/Aptfile
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get -yq dist-upgrade && \
  DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends \
    build-essential \
    postgresql-client-$PG_MAJOR \
    nodejs \
    yarn \
    $(cat /tmp/Aptfile | xargs) && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    truncate -s 0 /var/log/*log

First, let's talk about the Aptfile trick:

COPY .dockerdev/Aptfile /tmp/Aptfile
RUN apt-get install \
    $(cat /tmp/Aptfile | xargs)

I borrowed this idea from heroku-buildpack-apt, which allows installing additional packages on Heroku. If you're using this buildpack, you can even re-use the same Aptfile for local and production environment (though the buildpack's one provides more functionality).

Our default Aptfile contains only a single package (we use Vim to edit Rails Credentials):

vim

In one of the previous project I worked on, we generated PDFs using LaTeX and TexLive. Our Aptfile might look like this (those days I didn't use this trick):

vim
texlive
texlive-latex-recommended
texlive-fonts-recommended
texlive-lang-cyrillic

This way, we keep the task-specific dependencies in a separate file, making our Dockerfile more universal.

With regards to DEBIAN_FRONTEND=noninteractive, I kindly ask you to take a look at answer on Ask Ubuntu.

The --no-install-recommends switch helps us to save some space (and make our image slimmer) by not installing recommended packages. See more here.

The last part of this RUN (apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && truncate -s 0 /var/log/*log) also serves the same purpose—clears out the local repository of retrieved package files (we installed everything, we don't need them anymore) and all the temporary files and logs created during the installation. We need this cleanup to be in the same RUN statement to make sure this particular Docker layer doesn't contain any garbage.

The final part is mostly devoted to Bundler:

ENV LANG=C.UTF-8 \
  GEM_HOME=/bundle \
  BUNDLE_JOBS=4 \
  BUNDLE_RETRY=3
ENV BUNDLE_PATH $GEM_HOME
ENV BUNDLE_APP_CONFIG=$BUNDLE_PATH \
  BUNDLE_BIN=$BUNDLE_PATH/bin
ENV PATH /app/bin:$BUNDLE_BIN:$PATH

# Upgrade RubyGems and install required Bundler version
RUN gem update --system && \
    gem install bundler:$BUNDLER_VERSION

The LANG=C.UTF-8 sets the default locale to UTF-8. Otherwise Ruby uses US-ASCII for strings and bye-bye those sweet sweet emojis 👋

We set the path for gem installations via GEM_HOME=/bundle. What is /bundle? That's the path where we're going to mount as a volume later to persist the dependencies on the host system, i.e., your development machine (see below in docker-compose.yml).

The BUNDLE_PATH and BUNDLE_BIN variables tell Bundler where to look for gems and Ruby executables.

Finally, we expose Ruby and application binaries globally:

ENV PATH /app/bin:$BUNDLE_BIN:$PATH

That allows us to run rails, rake, rspec and other binstubbed commands without prefixing them with bundle exec.

`docker-compose.yml`

Docker Compose is a tool to orchestrate our containerized environment. It allows us to link containers to each other, define persistent volumes and services.

Below is the compose file for a typical Rails application development with PostgreSQL as a database, and Sidekiq background job processor:

version: '3.4'

services:
  app: &app
    build:
      context: .
      dockerfile: ./.dockerdev/Dockerfile
      args:
        RUBY_VERSION: '2.6.3'
        PG_MAJOR: '11'
        NODE_MAJOR: '11'
        YARN_VERSION: '1.13.0'
        BUNDLER_VERSION: '2.0.2'
    image: example-dev:1.0.0
    tmpfs:
      - /tmp

  backend: &backend
    <<: *app
    stdin_open: true
    tty: true
    volumes:
      - .:/app:cached
      - rails_cache:/app/tmp/cache
      - bundle:/bundle
      - node_modules:/app/node_modules
      - packs:/app/public/packs
      - .dockerdev/.psqlrc:/root/.psqlrc:ro
    environment:
      - NODE_ENV=development
      - RAILS_ENV=${RAILS_ENV:-development}
      - REDIS_URL=redis://redis:6379/
      - DATABASE_URL=postgres://postgres:postgres@postgres:5432
      - BOOTSNAP_CACHE_DIR=/bundle/bootsnap
      - WEBPACKER_DEV_SERVER_HOST=webpacker
      - WEB_CONCURRENCY=1
      - HISTFILE=/app/log/.bash_history
      - PSQL_HISTFILE=/app/log/.psql_history
      - EDITOR=vi
    depends_on:
      - postgres
      - redis

  runner:
    <<: *backend
    command: /bin/bash
    ports:
      - '3000:3000'
      - '3002:3002'

  rails:
    <<: *backend
    command: bundle exec rails server -b 0.0.0.0
    ports:
      - '3000:3000'

  sidekiq:
    <<: *backend
    command: bundle exec sidekiq -C config/sidekiq.yml

  postgres:
    image: postgres:11.1
    volumes:
      - .psqlrc:/root/.psqlrc:ro
      - postgres:/var/lib/postgresql/data
      - ./log:/root/log:cached
    environment:
      - PSQL_HISTFILE=/root/log/.psql_history
    ports:
      - 5432

  redis:
    image: redis:3.2-alpine
    volumes:
      - redis:/data
    ports:
      - 6379

  webpacker:
    <<: *app
    command: ./bin/webpack-dev-server
    ports:
      - '3035:3035'
    volumes:
      - .:/app:cached
      - bundle:/bundle
      - node_modules:/app/node_modules
      - packs:/app/public/packs
    environment:
      - NODE_ENV=${NODE_ENV:-development}
      - RAILS_ENV=${RAILS_ENV:-development}
      - WEBPACKER_DEV_SERVER_HOST=0.0.0.0

volumes:
  postgres:
  redis:
  bundle:
  node_modules:
  rails_cache:
  packs:

We define eight services. Why so many? Some of them only define shared configuration for others (abstract services, e.g., app and backend), others are used to specific commands using the application container (e.g., runner).

With this approach, we do not use docker-compose up command to run our application, but always specify the exact service we want to run (e.g., docker-compose up rails). That makes sense: in development, you rarely need all of the services up and running (Webpacker, Sidekiq, etc.).

Let's take a thorough look at each service.

`app`

The main purpose of this service is to provide all the required information to build our application container (the one defined in the Dockerfile above):

build:
  context: .
  dockerfile: ./.dockerdev/Dockerfile
  args:
    RUBY_VERSION: '2.6.3'
    PG_MAJOR: '11'
    NODE_MAJOR: '11'
    YARN_VERSION: '1.13.0'
    BUNDLER_VERSION: '2.0.2'

The context directory defines the build context for Docker: this is something like a working directory for the build process, it's used by the COPY command, for example.

We explicitly specify the path to Dockerfile since we do not keep it in the project root, packing all Docker-related files inside a hidden .dockerdev directory.

And, as we mentioned earlier, we specify the exact version of dependencies using args declared in the Dockerfile.

One thing that we should pay attention to is the way we tag images:

image: example-dev:1.0.0

One of the benefits of using Docker for development is the ability to synchronize the configuration changes across the team automatically. You only need to upgrade the local image version every time you make changes to it (or to the arguments or files it relies on). The worst thing you can do is to use example-dev:latest as your build tag.

Keeping an image version also helps to work with two different environments without any additional hassle. For example, when you work on a long-running "chore/upgrade-to-ruby-3" branch, you can easily switch to master and use the older image with the older Ruby, no need to rebuild anything.

The worst thing you can do is to use latest tags for images in your docker-compose.yml.

We also tell Docker to use tmpfs for /tmp folder within a container to speed things up:

tmpfs:
  - /tmp

`backend`

We reached the most interesting part of this post.

This service defines the shared behavior of all Ruby services.

Let's talk about the volumes first:

volumes:
  - .:/app:cached
  - bundle:/bundle
  - rails_cache:/app/tmp/cache
  - node_modules:/app/node_modules
  - packs:/app/public/packs
  - .dockerdev/.psqlrc:/root/.psqlrc:ro

The first item in the volumes list mounts the current working directory (the project's root) to the /app folder within a container using the cached strategy. This cached modifier is the key to efficient Docker development on MacOS. We're not going to dig deeper in this post (we're working on a separate one on this subject 😉), but you can take a look at the docs.

The next line tells our container to use a volume named bundle to store /bundle contents. This way we persist our gems data across runs: all the volumes defined in the docker-compose.yml stay put until we run docker-compose down --volumes.

The following three lines are also there to get rid of the "Docker is slow on Mac" curse. We put all the generated files into Docker volumes to avoid heavy disk operations on the host machine:

- rails_cache:/app/tmp/cache
- node_modules:/app/node_modules
- packs:/app/public/packs

To make Docker fast enough on MacOS follow these two rules: use :cached to mount source files and use volumes for generated content (assets, bundle, etc.).

The last line adds a specific psql configuration to the container. We mostly need it to persist the commands history by storing it in the app's log/.psql_history file. Why psql in the Ruby container? It's used internally when you run rails dbconsole.

Our .psqlrc file contains the following trick to make it possible to specify the path to the history file via the env variable (allow specifying the path to history file via PSQL_HISTFILE env variable, and fallback to the defaukt $HOME/.psql_history otherwise):

\set HISTFILE `[[ -z $PSQL_HISTFILE ]] && echo $HOME/.psql_history || echo $PSQL_HISTFILE`

Let's talk about the environment variables:

environment:
  - NODE_ENV=${NODE_ENV:-development}
  - RAILS_ENV=${RAILS_ENV:-development}
  - REDIS_URL=redis://redis:6379/
  - DATABASE_URL=postgres://postgres:postgres@postgres:5432
  - WEBPACKER_DEV_SERVER_HOST=webpacker
  - BOOTSNAP_CACHE_DIR=/bundle/bootsnap
  - HISTFILE=/app/log/.bash_history
  - PSQL_HISTFILE=/app/log/.psql_history
  - EDITOR=vi
  - MALLOC_ARENA_MAX=2
  - WEB_CONCURRENCY=${WEB_CONCURRENCY:-1}

There are several things here, and I'd like to focus one.

First, the X=${X:-smth} syntax. It could be translated as "For X variable within the container use the host machine X env variable value if present and another value otherwise". Thus, we make it possible to run a service in a different environment provided along with the command, e.g., RAILS_ENV=test docker-compose up rails.

The DATABASE_URL, REDIS_URL, and WEBPACKER_DEV_SERVER_HOST variables connect our Ruby application to other services. The DATABASE_URL and WEBPACKER_DEV_SERVER_HOST variables are supported by Rails (ActiveRecord and Webpacker respectively) out-of-the-box. Some libraries support REDIS_URL as well (Sidekiq) but not all of them (for instance, Action Cable must be configured explicitly).

We use bootsnap to speed up the application load time. We store its cache in the same volume as the Bundler data because this cache mostly contains the gems data; thus, we should drop everything altogether in case we do another Ruby version upgrade, for instance.

The HISTFILE=/app/log/.bash_history is the significant setting from the developer's UX point of view: it tells Bash to store its history in the specified location, thus making it persistent.

The EDITOR=vi is used, for example, by rails credentials:edit command to manage credentials files.

Finally, the last two settings, MALLOC_ARENA_MAX and WEB_CONCURRENCY, are there to help you keep Rails memory handling in check.

The only lines in this service yet to cover are:

stdin_open: true
tty: true

They make this service interactive, i.e., provide a TTY. We need it, for example, to run Rails console or Bash within a container.

It is the same as running a Docker container with the -it options.

`webpacker`

The only thing I want to mention here is the WEBPACKER_DEV_SERVER_HOST=0.0.0.0 setting: it makes Webpack dev server accessible from the outside (by default it runs on localhost).

`runner`

To explain what is this service for, let me share the way I use Docker for development:

I start a Docker daemon running a custom docker-start script:

#!/bin/sh

if ! $(docker info > /dev/null 2>&1); then
  echo "Opening Docker for Mac..."
  open -a /Applications/Docker.app
  while ! docker system info > /dev/null 2>&1; do sleep 1; done
  echo "Docker is ready to rock!"
else
  echo "Docker is up and running."
fi

Then I run dcr runner (dcr is an alias for docker-compose run) in the project directory to log into the container's shell; this is an alias for:

$ docker-compose run --rm runner

I run (almost) everything from within this container: tests, migrations, Rake tasks, whatever.

As you can see, I do not spin a new container every time I need to run a task, and I'm always using the same one.

Thus, I'm using dcr runner the same way I used vagrant ssh years ago.

The only reason why it's called runner and not shell, for example, is that it also could be used to run arbitrary commands within a container.

Note: The runner service is a matter of taste, it doesn't bring anything new comparing to the web service, except from the default command (/bin/bash); thus, docker-compose run runner is exactly the same as docker-compose run web /bin/bash (but shorter 😉).

Bonus: dip.yml

If you still think that the Docker Compose way is too complicated, there is a tool called Dip developed by one of my colleages at Evil Martians that aims to make the developer experience smoother.

It is especially useful if you have multiple compose files or platform-dependent configurations because it could glue them together and provide a universal interface to manage the Docker development environment.

We're going to tell you more about it in the future. Stay tuned!

P.S. Special thanks to Sergey Ponomarev and Mikhail Merkushin for sharing their tips on the subject. 🤘

Read more dev articles on https://evilmartians.com/chronicles!

Reporting non-nullable violations in graphql-ruby properly

Andrey Novikov — Fri, 19 Jul 2019 07:37:08 +0000

GraphQL encourages you to decouple your frontend from your backend. You can think more about your types’ “correctness” and less about particular view layout. To achieve that GraphQL provides strong type system. GraphQL doesn’t allow to return String instead of Integer and also doesn’t allow return nulls where API declares that type is non-nullable.

But sometimes your data will accidentally contain empty values where even you don’t expect them. Some external APIs will return incorrect or missing data, even if its contract disallows that. Sometimes it will be your own bug. But in any case, it is your responsibility to track for such failures and correct them.

If you use Ruby for developing GraphQL API, the chances are high that you use graphql-ruby. It is standard de-facto in Ruby because it is a very mature and feature-rich gem with a whole ecosystem of plugins. But “out of the box” there it doesn’t inform you about violations of these “non-nullable type” rules and just silently returns a partial response with error messages to the client.

But we want to know about non-null violations because it is our responsibility to develop bullet-proof applications. And according to the graphql-ruby’s documentation about type errors, there is a special hook to catch such errors. Let’s use it!

We’re using Honeybadger, so we’ll report this offense there:

class YourAppSchema < GraphQL::Schema
  def self.type_error(exception, query_context)
    if exception.is_a?(::GraphQL::InvalidNullError)
      Honeybadger.notify exception, context: { query: query_context.query.query_string }
    end
    super
  end
end

So far, so good. Soon we’ll able to see our first error in error tracker UI.

But wait, what’s it? There are errors about non-null violations in multiple different fields mixed in one error field. Aren’t they different bugs? What’s wrong?

It turns out that by default Honeybadger groups errors together by exception class, component (controller and action), and backtrace fragment. Please note that the exception message doesn’t count. In the case of GraphQL, it is always a single controller and action, so exceptions for absolutely different queries are getting grouped together. Let’s fix that!

According to the same documentation page, we need to calculate unique fingerprint string for every different error. So, what we need to include to this fingerprint for non-null violation?

Exception class? Yes, we need to distinguish GraphQL::InvalidNullError from other types of errors
Controller and action? No, they’re always the same. And even if an error will occur during triggering subscription, will it make any difference?
Backtrace? Hmm… I’m not sure. What do you think?
Type and field names, of course! We need to separate errors in different fields.

Let’s use an exception class name, GraphQL type name, and field name:

class ApplicationSchema < GraphQL::Schema
  class << self
    def type_error(exception, query_context)
      fingerprint = honeybadger_fingerprint(exception)
      Honeybadger.notify exception, fingerprint: fingerprint, context: {
        query:   query_context.query.query_string,
        user_id: query_context[:user]&.id,
      }
      super
    end

    private

    def honeybadger_fingerprint(error)
      case error
      when GraphQL::InvalidNullError
        "#{error.class}:#{error.parent_type.name}.#{error.field.name}"
      end
    end
  end
end

And that’s it: from now on exceptions for different fields lives in different Honeybadger errors.

Thank you for your attention! Let’s build reliable APIs!

DEV Community: Evil Martians

Nano Stores in Angular: how to make the state management simpler

What do we have now?

RxJS

NgRx

Nano Stores

Why is Nanostores the handiest way to manage state?

Climbing Steep hills, or adopting Ruby 3 types

RBS in a nutshell

Getting started with RBS

Introducing Steep

steep init

steep stats

Fighting with signatures, or make steep check happy

1. Refinements always break things.

2. Superclass don't cry 😢

3. Explicit over implicit: handling splats.

4. Explicit over implicit, pt. 2: forwarding arguments.

5. Variadic arguments: annotations to the rescue!

6. Deal with metaprogramming: interfaces.

7. Making Steep happy.

Runtime type checking with RBS

Bonus: Steep meets Rake

Bonus 2: Type Checking meets GitHub Actions

RuboCoping with legacy

Style matters

TODO or not TODO

You shall not pass: introducing .rubocop_strict.yml

One Standard to rule them all

Beyond the Standard

Pulling the trigger: How to update counter caches in your Rails app without Active Record callbacks

Trigger finger

Off to the races

Lock-free alternative

Loop instead of UPSERT

Persisted queries in GraphQL: Slim down Apollo requests to your Ruby application

GitHub Actions: First impressions

Warming up: dealing with the stale issues

Migrating Markdown linters

Adding Neo and Trinity for RSpec

Dealing with JRuby

Bonus: multiple badges

Keeping OSS documentation with Docsify, Lefthook, and friends

In the Beginning, There Was Chaos

GitHub Wiki

docs folder

Jekyll & GitHub Pages

Docusaurus

Docsifying documentation

Keeping docs healthy with linters

Better web video with AV1 codec

Codecs and Containers

Meet AV1

How to use AV1 right now

Understand compression options

Play nice with browsers

Bonus: how to convert GIFs to AV1 and H.264

Fullstaq Ruby: First impressions, and how to migrate your Docker/Kubernetes Ruby apps today

What is Fullstaq Ruby?

Why do we need it?

Now that's the difference!

Alternatives?

Cables vs. malloc_trim, or yet another Ruby memory usage benchmark

Vladimir Dementyev ・ Mar 19 '19 ・ 6 min read

How to install?

Recap

Speeding up Go Modules for Docker and CI

Caching on CI

Caching in Docker

In Development

In production

In a nutshell

Ruby on Whales: Dockerizing Ruby and Rails development

Dockerfile

docker-compose.yml

app

backend

webpacker

runner

Bonus: dip.yml

`steep init`

`steep stats`

Fighting with signatures, or make `steep check` happy

You shall not pass: introducing `.rubocop_strict.yml`

`docs` folder

`Dockerfile`

`docker-compose.yml`

`app`

`backend`

`webpacker`

`runner`